summaryrefslogtreecommitdiff
path: root/plugins
AgeCommit message (Collapse)Author
2021-07-11Update JOBS interface, Renamed Consume -> Run.Valery Piashchynski
Add DYNAMIC declaration of the pipelines. Update Jobs constructor interface, add FromPipeline method to construct jobs driver (unique) via the `Declare` RPC call. Add `Stop` method to gracefully stop all consumers. Binary heaps `GetMax` to canonical `ExtractMin`. Other small improvements. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-11Dead letter exchange optimization. Cache for the DLX queues. If theValery Piashchynski
queue had been declared and binded to the particular timeout, we can avoid re-declaring the queue. This optimization increases RPS to the DLX from 3.5k to 35k. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-11unpack: use fast string convertValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-11Update informer interface to return slice of pointers (do not over-copyValery Piashchynski
the Stat structure). Make amqp Push concurrent safe. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10Return structure instead of interface in places where that possibleValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10SQS skeletonValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10DeliveryMode -> amqp.Persistent for all Publish operationsValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10DeliveryMode -> amqp.PersistentValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10Update Consumer interface, List method returns []string of theValery Piashchynski
pipelines. Update packing and unpacking Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10Properly parse amqp items (jobs).Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-10AMQP job driver...Valery Piashchynski
Update main driver's interface, add Consume(*pipeline) method. Implement it on the amqp and ephemeral drivers. Fix error with incorrect order of Register <-> Consume method calls. Implement rabbitMQ driver, add timeouts, dead-letter-exchange, packing-unpacking of the amqp messages. Implement AMQP redialer in case of network error as well as channels re-creation. Update drawio diagram. Update .rr.yaml jobs configuration, add all amqp options. Implement Ack/Nack. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08AMQP update...Valery Piashchynski
Add redialer, consumer, rabbit queues initializer. Update configuration (.rr.yaml). Add ack/nack for the jobs main plugin with error handling. Add Qos, queues bining and declaration. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08Remove duplicated test-helpersValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08Add pipeline and job plugin options...Valery Piashchynski
Skeleton for the amqp plugin. Add Timeout and Pipeline to the job.Context() method. Implement queue limits for the ephemeral driver with main priority queue limits. Update configuration, add pipeline_size for every pipeline and jobs priority queue size. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-08Add length limiter for the binary heaps data structure. After max lengthValery Piashchynski
reached, insert operation will be blocked until all pending messages have been fully consumed. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-07New Methods in the binary heap interface...Valery Piashchynski
Add Len() method to the Binary Heaps interface with implementation. Start consumers only for the user-defined set from the config. Add Headers field to the proto Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-07Updated ephemeral plugin, PQ and protobuf...Valery Piashchynski
Implement core of the root jobs plugin with a proper drivers/pipelines handling mechanism. Add delayed jobs for the ephemeral plugin. Remove ResumeAll, Resume, StopAll, Stop. Replaced with Pause/Resume with a slice of the pipelines. Other small improvements. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-06Update Jobs interface...Valery Piashchynski
Use bh.len everywhere in the binary heaps algo instead of direct len check. Add Ack/Nack to the main jobs loop. Add PushBatch method to the jobs rpc layer. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-06Rework ephemeral and binary heapsValery Piashchynski
Implemented a sync.Cond for binary heap algo to save processor from spinning in the for loop and receiving nil Items until the Queue will be filled. Add num_pollers option to the configuration to specify number of pollers from the queue. Add Resume, ResumeAll, Stop, StopAll, PushBatch methods to the ephemeral. Remove map and use sync.Map in the ephemeral broker. Add protobuf schema. Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-05- Update PQValery Piashchynski
- Update ephemeral plugin, complete Push - Add Jobs full configuration Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-05- Finish binary_heapValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-03- Initial binary_heapValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-02- joinTopics/Server as constants, do not allocate strings on everyValery Piashchynski
request Signed-off-by: Valery Piashchynski <[email protected]>
2021-07-01- Remove DispatcherValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-30- Update arch diagramsValery Piashchynski
- Update ephemeral plugin Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-25- Update jobs interfaceValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-25Merge remote-tracking branch 'origin/master' into feature/jobs_pluginValery Piashchynski
2021-06-24- Clear BOLTDB TTL memory region on Clear RPC callValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-24- Add Clear method to the storagesValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-23- Add PQ (priority_queue) mockValery Piashchynski
- Add binary heap mock - Connect first sub-plugin (ephemeral) with root jobs plugin Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-23- Swithc from value to pointer to the pool configuration. InterfaceValery Piashchynski
value changed. Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-23- Update arch, pluignValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-22- Folders structValery Piashchynski
- Initial ephemeral broker commit - Initial RPC Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-22- Move common interfaces and structures to the 'common' folderValery Piashchynski
- Update tests Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-22- Remove all old code, reformat, fix linters, return GAValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-22- Temporary turn off CIValery Piashchynski
- Remove old PHP tests Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-21- Rework dispatcher, pipeline, job (not completely)Valery Piashchynski
Create a config sample with RR2 support. Progress on root JOBS plugin. Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-21Merge remote-tracking branch 'origin/master' into feature/jobs_pluginValery Piashchynski
# Conflicts: # plugins/websockets/plugin.go
2021-06-21- Minor tests improvenmentsValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-20- Improve error messagesValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-20- Update redis/memory pubsubsValery Piashchynski
- Rework internal message bus - Add new tests for the broadcast plugin and include them into the GA Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-18- TyposValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-18- Add broadcast testsValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-18- Rework finishedValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-18- Rework main partsValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-17- Initial broadcast commitValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-16- Update ws plugin responses (text instead of binary)Valery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-16- Update CHANGELOGValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-16- Add origin check for the websocketsValery Piashchynski
Signed-off-by: Valery Piashchynski <[email protected]>
2021-06-16- Update jobs sourcesValery Piashchynski
- Update Arch diagramm Signed-off-by: Valery Piashchynski <[email protected]>
generated by cgit v1.2.3 (git 2.25.1) at 2025-01-26 07:14:40 +0000