summaryrefslogtreecommitdiff
path: root/plugins/jobs/plugin.go
AgeCommit message (Collapse)Author
2021-09-16Move plugins to a separate repositoryValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-09-02Do not use copied driver from the map.Valery Piashchynski
Store driver and pipeline only after run w/o error. Signed-off-by: Valery Piashchynski <[email protected]>
2021-09-02Update endure.Valery Piashchynski
Replace map with sync.Map because various RPC methods can be called simultaneously. Signed-off-by: Valery Piashchynski <[email protected]>
2021-09-01Remove redundand debug logValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-09-01Close connection in the amqp driver.Valery Piashchynski
bytes.Buffer update in the beanstalk driver Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-31Naming: service -> pluginValery Piashchynski
Fix bug with survived workers in the debug mode Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-31Tune listener timersValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-30Add error to the EventJobError eventValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-30Tests for the boltdb jobs.Valery Piashchynski
Fix issue with Stop in the jobs plugin which didn't destroy the pool. Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-29Implement Init, FromPipeline methodsValery Piashchynski
Update receiver in the amqp driver Add simple (initial) boltdb tests Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-18Add prometheus metrics for the jobs, update testsValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-14Update broadcast tests, add redis flusing. Initial impl of the jobValery Piashchynski
drivers state. Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-13Update go.mod dependencies to the most recent version.Valery Piashchynski
Fix tests according to the new mocks. Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-12Remove unneeded options, complete tests for the ephemeral, update protoValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-12Fix various bugs in the SQS. Implement SQS tests for theValery Piashchynski
jobs_ok.php/jobs_err.php workers. Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-11Durable requeue algo. Update AMQP and Beanstalk tests to use mockValery Piashchynski
logger. Fix bugs discovered during testing. Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-11Increase timeout to 60s.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-10Add jobs protocol support for the AMQP driverValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-09Add initial support for the php worker protocol.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-08-09Update goridge to v3.2.0, update all frames operations.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-27Initial support for the cancellation via contextValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-23EventPipeRun -> EvenPipeActiveValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-22Fix AMQP bugs, add more amqp testsValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-22Format-edValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-22Passing by ptr is slightly faster:Valery Piashchynski
Benchmark_Ptr Benchmark_Ptr-32 907456771 1.354 ns/op 0 B/op 0 allocs/op Benchmark_Value Benchmark_Value-32 791122101 1.366 ns/op 0 B/op 0 allocs/o In terms of OPS. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-22BC for the Pool, worker interfaces, pass/return payload by pointerValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-22Update consumers, tests stabilizationValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-21Fix AMQP context, add ID, job, other fields.Valery Piashchynski
Fix sqs queue re-creation. Complete redia for the beanstalk. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-18Started beanstalk driver. Add new Queue impl (not finished yet).Valery Piashchynski
Fix bugs in the AMQP, update proto-api Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-15Remove duplicated loggingValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-14SQS driver.Valery Piashchynski
Fix isssues in the AMQP driver. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-12Add JOBS tests to the Makefile and GitHub CI.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-12Update JOBS interface, remove List() method, implemented on the root RPCValery Piashchynski
level. AMQP consumer replace sync.Map with atomic.Value, because we associate only 1 pipeline with a driver. So, we can store pipeline in the atomic.Value. Implement events handler, add job events. Use job events to push information to the logger. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-12Typo in the pool configuration.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-12Implement Reset operation.Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-11Update JOBS interface, Renamed Consume -> Run.Valery Piashchynski
Add DYNAMIC declaration of the pipelines. Update Jobs constructor interface, add FromPipeline method to construct jobs driver (unique) via the `Declare` RPC call. Add `Stop` method to gracefully stop all consumers. Binary heaps `GetMax` to canonical `ExtractMin`. Other small improvements. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-11Update informer interface to return slice of pointers (do not over-copyValery Piashchynski
the Stat structure). Make amqp Push concurrent safe. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10AMQP job driver...Valery Piashchynski
Update main driver's interface, add Consume(*pipeline) method. Implement it on the amqp and ephemeral drivers. Fix error with incorrect order of Register <-> Consume method calls. Implement rabbitMQ driver, add timeouts, dead-letter-exchange, packing-unpacking of the amqp messages. Implement AMQP redialer in case of network error as well as channels re-creation. Update drawio diagram. Update .rr.yaml jobs configuration, add all amqp options. Implement Ack/Nack. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08AMQP update...Valery Piashchynski
Add redialer, consumer, rabbit queues initializer. Update configuration (.rr.yaml). Add ack/nack for the jobs main plugin with error handling. Add Qos, queues bining and declaration. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08Remove duplicated test-helpersValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08Add pipeline and job plugin options...Valery Piashchynski
Skeleton for the amqp plugin. Add Timeout and Pipeline to the job.Context() method. Implement queue limits for the ephemeral driver with main priority queue limits. Update configuration, add pipeline_size for every pipeline and jobs priority queue size. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08Add length limiter for the binary heaps data structure. After max lengthValery Piashchynski
reached, insert operation will be blocked until all pending messages have been fully consumed. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-07New Methods in the binary heap interface...Valery Piashchynski
Add Len() method to the Binary Heaps interface with implementation. Start consumers only for the user-defined set from the config. Add Headers field to the proto Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-07Updated ephemeral plugin, PQ and protobuf...Valery Piashchynski
Implement core of the root jobs plugin with a proper drivers/pipelines handling mechanism. Add delayed jobs for the ephemeral plugin. Remove ResumeAll, Resume, StopAll, Stop. Replaced with Pause/Resume with a slice of the pipelines. Other small improvements. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-06Update Jobs interface...Valery Piashchynski
Use bh.len everywhere in the binary heaps algo instead of direct len check. Add Ack/Nack to the main jobs loop. Add PushBatch method to the jobs rpc layer. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-06Rework ephemeral and binary heapsValery Piashchynski
Implemented a sync.Cond for binary heap algo to save processor from spinning in the for loop and receiving nil Items until the Queue will be filled. Add num_pollers option to the configuration to specify number of pollers from the queue. Add Resume, ResumeAll, Stop, StopAll, PushBatch methods to the ephemeral. Remove map and use sync.Map in the ephemeral broker. Add protobuf schema. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-05- Update PQValery Piashchynski
- Update ephemeral plugin, complete Push - Add Jobs full configuration Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-03- Initial binary_heapValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-01- Remove DispatcherValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-30- Update arch diagramsValery Piashchynski
- Update ephemeral plugin Signed-off-by: Valery Piashchynski <[email protected]>
generated by cgit v1.2.3 (git 2.25.1) at 2025-01-24 07:30:40 +0000