I have created a Tool using Scala to improve the speed of spark executions with very good results, and in my opinion with some little modifications could be apply to any Scala applications.
The idea of the tool is for the application to: detect, sort and execute all the Methods in parallel, been aware of dependencies and critical path.
Changes: the original version of the tool is designed to works in spark, that means that every method will return DataFrames, but also that because Spark removes the DataFrames we can’t use the futures and promises to safely transfer DataFrames between methods (even if we try to cache the DataFrame).
If this tool is use for Scala we can simplify the tool not storing the information and using futures and promises to transfer the result of the methods
If someone want to look the current code of the tool:
Do you think is a good idea? Will be used by the Scala community? Add as a tool or as part of the language?