PyDataSentry - Memory for Data Science¶
pydatasentry package allows auditability of modeling code and data by logging all relevant information for every single model run (e.g., a regression)
Background¶
Audience
Mainly data scientists that run statistical models using pandas/statsmodels/scikit-learn of various kinds on a regular basis.
Problem
The combination of datasets, questions, and nature of analysis is growing everyday. Data scientists find it hard to keep track of all the different datasets they dealt with, what they did with those datasets, and what they presented to the model-audience (business etc)
Solution
pydatasentry package allows auditability of modeling code and data by logging all relevant information for every single model run (e.g., a regression) You could use this for audit past results for correctness, share models and results with peers, search past results to avoid repition of work.
Features¶
- Automatic interception of the library calls. Right now only statsmodels.formula.api is supported. But the support can be easily extended to other libraries
- Capture of all relevant context for each run including the signature (who called, with what parameters etc.), the input dataset, the resulting object (full and summary versions). In addition parameters (sentryopt) can be passed to the library call that is extracted and captured (e.g., a set of tags)
- Storage in local directory in systematic way locally and remotely if needed.
- Optional github commit information to know what code was responsible for this call.
And all these can be over-ridden and extended.
Installation¶
pip install glob2 numpy scipy patsy scipy six statsmodels pandas
pip install pydatasentry
We need explicit install for now because scikit-learn has numpy dependency but setuptools is not installing dependencies in the order that we are specifying.
Examples¶
No code change:
$ sentry.py
<see the help>
sentry: Transparently instrument pandas code
sentry.py help
sentry.py init <sentry-conf.py>
sentry.py run [-c|--config <sentry-conf.py>] <python-program-to-be-instrumented>
$ sentry.py example basic_ols.py
<creates a simple OLS regression file>
$ sentry.py init sentry-conf.py
$ emacs sentry-conf.py
<initialize a sentry configuration file and edit>
<update scope and run>
$ sentry.py run -c sentry-conf.py basic_ols.py
$ find model-output
<see the output>
Minimal:
Only two lines are required by default. Please check the config module to know what are the defaults for what needs to be captured (dataframes, statsmodels interface, signature) and where they should be stored (local directory ‘model-output’)
#!/usr/bin/env python
import os, sys
import pandas as pd
import statsmodels.formula.api as smf
import pydatasentry
if __name__ == "__main__":
pydatasentry.initialize()
df = pd.DataFrame({"A": [10,20,30,40,50],
"B": [20, 30, 10, 40, 50],
"C": [32, 234, 23, 23, 42523]})
result = smf.ols(formula="A ~ B + C",
data=df
).fit()
print(result.summary())
The output is stored in a experiment and time dependent directory that has a unique identifier associated with it.
$ find model-output
model-output
model-output/offers
model-output/offers/conditional
model-output/offers/conditional/1
model-output/offers/conditional/1/ols
model-output/offers/conditional/1/ols/96f5b468-85ee-11e5-b3b5-0800274d1e8c
model-output/offers/conditional/1/ols/96f5b468-85ee-11e5-b3b5-0800274d1e8c/2015-Nov-08-13:29:08
model-output/offers/conditional/1/ols/96f5b468-85ee-11e5-b3b5-0800274d1e8c/2015-Nov-08-13:29:08/full.pickle
model-output/offers/conditional/1/ols/96f5b468-85ee-11e5-b3b5-0800274d1e8c/2015-Nov-08-13:29:08/signature.json
model-output/offers/conditional/1/ols/96f5b468-85ee-11e5-b3b5-0800274d1e8c/2015-Nov-08-13:29:08/summary.pickle
$ cat model-output/offers/conditional/1/ols/96f5b468-85ee-11e5-b3b5-0800274d1e8c/2015-Nov-08-13:29:08/signature.json
{
"data": {
"name": "random",
"columns": [
"A",
"B",
"C"
],
"shape": [
5,
3
]
},
"uuid": "51ef2ae4-85ed-11e5-a8bc-0800274d1e8c",
"model": {
"module": "statsmodels.formula.api",
"formula": "A ~ B + C",
"function": "ols"
},
"experiment": {
"scope": "test",
"version": 1,
"run": "test"
}
}
Detailed:
pydatasentry gives the user control over every aspect of the process. The example below shows the user over-riding the experiment details, output parameters, and tracking lineage.
#!/usr/bin/env python
import os, sys
import pandas as pd
import statsmodels.formula.api as smf
import pydatasentry
if __name__ == "__main__":
# Specify what and how of the capture in great detail
pydatasentry.initialize({
'debug': True,
'spec': {
'experiment': {
'scope': 'test',
'run': 'test',
'version': 1
},
'output': {
'params': [
{
'content': 'attributes.output.default-signature',
'path': 'attributes.output.relative-path',
'filename': 'signature.json'
}
]
},
},
})
with tracklineage("load", "sample"):
df = pd.DataFrame({"A": [10,20,30,40,50],
"B": [20, 30, 10, 40, 50],
"C": [32, 234, 23, 23, 42523]})
result = smf.ols(formula="A ~ B + C",
data=df,
sentryopts={
'dataset': "sample"
}
).fit()
print(result.summary())
Caveats¶
- Only python3 is supported.
Next Steps¶
A number of next steps are planned:
- Test will several statsmodels libraries
- Improve the instrumentation.
Please let me (pingali@gmail.com) know or post an issue
License¶
Standard MIT License. See LICENSE.txt
Acknowledgements¶
To FourthLion for agreeing to contribute this code back to the community.