Automating Processes with Spark and Python

Hey everyone, Tsun here again to talk about another use case for Spark. Today, we're going to explore how to use Spark and Python together to automate a process.

Example Model: Required Minimum Distribution Calculation

We'll be using an example model called the Required Minimum Distribution (RND) Calc. This model requires inputs such as:

• Years

• Date of birth for the beneficiary and the spouse

• Account balance

The outputs will be the current year RND amount and the modeled RND amount in the future.

Uploading and Converting to API

1. Tag your inputs and outputs as usual.

2. Upload the model into Spark by hitting "New Service" and dragging the RND calculation into Spark.

3. Spark will convert the spreadsheet's logic, calculations, formulas, and tables into fast C++ code.

4. The code is hosted on the cloud and made accessible via API.

Using Python to Call the API

Once the API is published, you can access the RND calculation anywhere in your system of choice. Here's how to use Python to call the API:

1. Access the API endpoint and submit your inputs to calculate the RND amount.

2. For large-scale operations, such as processing 1,000 policies, use Python to automate the process.

3. Read data from a SQL database, run calculations through the Spark API, and store results back into SQL.

Running the Python Script

Execute the Python script to automate the calculations:

1. Read policy data from the SQL database.

2. Run calculations through the Spark API.

3. Output results back into SQL, creating a new table with outputs for each policy.

Recap

This process involves converting Excel logic into an API and running it at scale. Python serves as an excellent orchestration tool to connect data to calculations. While this example focused on RND calculations, clients also use this method for data transformations, post-model processing, reporting, and reserve calculations. Spark provides full governance and auditability for each calculation.

Thanks for your time today!