summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2021-07-21Fix AMQP context, add ID, job, other fields.Valery Piashchynski
Fix sqs queue re-creation. Complete redia for the beanstalk. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-20Beanstalk driver alpha version ready.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-19Make beanstalk import not indirectValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-19Beanstalk initial commitValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-19Worker watcher interface update.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-18Remove channel from the QueueValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-18Started beanstalk driver. Add new Queue impl (not finished yet).Valery Piashchynski
Fix bugs in the AMQP, update proto-api Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-15Remove duplicated loggingValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14Update SQS item.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14Fix early returnValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14NPE when stopping RMQ channels.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14Fix incorrect path in the CI. Implement FromPipeline for the sqs.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14Merge masterValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14Merge remote-tracking branch 'origin/master' into feature/jobs_pluginValery Piashchynski
# Conflicts: # pkg/worker_watcher/interface.go
2021-07-14Update CHANGELOGValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14#749 bug(supervisor, ttl): worker gets into the inconsistent state after TTL ↵Valery Piashchynski
was reached #749 bug(supervisor, ttl): worker gets into the inconsistent state after TTL was reached
2021-07-14Fix milestone in the CHANGELOGValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14Update CHANGELOGValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14Fix TTL issue, added explanation comments.Valery Piashchynski
The worker after it executed the request, may overwrite the TTL state. This inconsistency leads to the +1 worker in the FIFO channel. In this state, the Push operation was blocked. Add RR_BROADCAST_PATH. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14Implement Pause/Resume/StopValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14SQS driver.Valery Piashchynski
Fix isssues in the AMQP driver. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-12Add JOBS tests to the Makefile and GitHub CI.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-12Return proper error type in the redialValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-12Update JOBS interface, remove List() method, implemented on the root RPCValery Piashchynski
level. AMQP consumer replace sync.Map with atomic.Value, because we associate only 1 pipeline with a driver. So, we can store pipeline in the atomic.Value. Implement events handler, add job events. Use job events to push information to the logger. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-12Typo in the pool configuration.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-12Implement Reset operation.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-12Finish dynamic declaration of the pipelines. Fix issue withValery Piashchynski
configuration parsing in the AMQP consumer. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-11Update JOBS interface, Renamed Consume -> Run.Valery Piashchynski
Add DYNAMIC declaration of the pipelines. Update Jobs constructor interface, add FromPipeline method to construct jobs driver (unique) via the `Declare` RPC call. Add `Stop` method to gracefully stop all consumers. Binary heaps `GetMax` to canonical `ExtractMin`. Other small improvements. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-11Dead letter exchange optimization. Cache for the DLX queues. If theValery Piashchynski
queue had been declared and binded to the particular timeout, we can avoid re-declaring the queue. This optimization increases RPS to the DLX from 3.5k to 35k. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-11unpack: use fast string convertValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-11Update informer interface to return slice of pointers (do not over-copyValery Piashchynski
the Stat structure). Make amqp Push concurrent safe. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10Move interfaces to its consumersValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10Return structure instead of interface in places where that possibleValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10SQS skeletonValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10DeliveryMode -> amqp.Persistent for all Publish operationsValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10DeliveryMode -> amqp.PersistentValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10Update Consumer interface, List method returns []string of theValery Piashchynski
pipelines. Update packing and unpacking Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10Properly parse amqp items (jobs).Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10Merge branch 'master' into feature/jobs_pluginValery Piashchynski
2021-07-10AMQP job driver...Valery Piashchynski
Update main driver's interface, add Consume(*pipeline) method. Implement it on the amqp and ephemeral drivers. Fix error with incorrect order of Register <-> Consume method calls. Implement rabbitMQ driver, add timeouts, dead-letter-exchange, packing-unpacking of the amqp messages. Implement AMQP redialer in case of network error as well as channels re-creation. Update drawio diagram. Update .rr.yaml jobs configuration, add all amqp options. Implement Ack/Nack. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08AMQP update...Valery Piashchynski
Add redialer, consumer, rabbit queues initializer. Update configuration (.rr.yaml). Add ack/nack for the jobs main plugin with error handling. Add Qos, queues bining and declaration. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08Update feature_request.mdValery Piashchynski
2021-07-08Remove duplicated test-helpersValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08maxLen for the binary heaps should be more than 0Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08Add pipeline and job plugin options...Valery Piashchynski
Skeleton for the amqp plugin. Add Timeout and Pipeline to the job.Context() method. Implement queue limits for the ephemeral driver with main priority queue limits. Update configuration, add pipeline_size for every pipeline and jobs priority queue size. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08Unlock binary heaps max length in the consume-rate testValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08Add length limiter for the binary heaps data structure. After max lengthValery Piashchynski
reached, insert operation will be blocked until all pending messages have been fully consumed. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-07New Methods in the binary heap interface...Valery Piashchynski
Add Len() method to the Binary Heaps interface with implementation. Start consumers only for the user-defined set from the config. Add Headers field to the proto Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-07Updated ephemeral plugin, PQ and protobuf...Valery Piashchynski
Implement core of the root jobs plugin with a proper drivers/pipelines handling mechanism. Add delayed jobs for the ephemeral plugin. Remove ResumeAll, Resume, StopAll, Stop. Replaced with Pause/Resume with a slice of the pipelines. Other small improvements. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-07Rework test env...Valery Piashchynski
Move all dockerfiles, docker-compose files into the env folder inside the tests folder. Add elasticmq dockerfile to build SQS compatible test env. Add beanstalk, rabbitmq, sqs to the main tests docker-compose file. Signed-off-by: Valery Piashchynski <[email protected]>
generated by cgit v1.2.3 (git 2.25.1) at 2025-01-24 19:54:49 +0000