WIP: source: Mysql Source feature #114

sudharsana-kjl · 2019-07-08T02:02:16Z

This PR hopes to add a feature to work with data from a mysql database.

dffml/source/mysql_source.py

CHANGELOG.md

johnandersen777 · 2019-07-09T16:56:49Z

import unittest

from dffml.util.testing.source import SourceTest
from dffml.util.asynctestcase import AsyncTestCase

from dffml.source.mysql import MysqlSourceConfig, MysqlSource


class TestMysqlSource(SourceTest, AsyncTestCase):
    async def setUpSource(self):
        return MysqlSource(
            MysqlSourceConfig(... parameters, get from os.getenv ...)
            # user=os.getenv("MYSQL_USER")
            # password=os.getenv("MYSQL_PASSWORD")
        )

    @unittest.SkipIf(os.getenv("MYSQL_USER", default=None) is None, "Need MYSQL_ env variables to run MySQL Source tests")
    async def test_update():
        await super().test_update()

We will start a mysql database within travis CI and set environment variables the same way we start the database, aka, same MYSQL_USER, MYSQL_PASSWORD from docker hub docs: @pdxjohnny will do this for travis.

dffml/source/mysql_source.py

johnandersen777 · 2019-07-09T17:06:20Z

To have aiomysql installed for testing, add to setup.py

    # ...
    packages=find_packages(),
    install_requires=[
        "aiomysql>=Go find the latest version number of PyPi and put it here",
    ],
    include_package_data=True,
    # ...

codecov-io · 2019-07-10T06:56:19Z

Codecov Report

Merging #114 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #114   +/-   ##
=======================================
  Coverage   92.79%   92.79%           
=======================================
  Files          57       57           
  Lines        3537     3537           
  Branches      408      408           
=======================================
  Hits         3282     3282           
  Misses        198      198           
  Partials       57       57

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2a459b3...39512c3. Read the comment docs.

…po query

dffml/source/mysql_source.py

johnandersen777 · 2019-07-14T18:13:07Z

dffml/source/mysql_source.py

+                yield await self.repo(src_url)
+
+    async def repo(self, src_url: str):
+        if self.parent.config.repo_query != "":


repo_query is required, so you don't need to check if it's an empty string

dffml/source/mysql_source.py

johnandersen777 · 2019-07-15T16:39:17Z

tests/source/test_mysql.py

+                table="repo_data",
+                repo_query="",
+                update_query="",
+                repos_query="",


This won't work unless you set repo_query, update_query, repos_query

dffml/source/mysql_source.py

Signed-off-by: John Andersen <[email protected]>

sudharsana-kjl · 2019-07-21T23:37:41Z

For testing purpose, SSL certificate was generated using openssl.

openssl genrsa 2048 > private.pem
openssl req -x509 -days 1000 -new -key private.pem -out ca.pem

lgtm-com · 2019-08-01T18:19:20Z

This pull request introduces 1 alert when merging c791b2c into 26e8313 - view on LGTM.com

new alerts:

1 for Unused import

lgtm-com · 2019-08-05T17:45:08Z

This pull request introduces 1 alert when merging 6c67cee into d4b2a35 - view on LGTM.com

new alerts:

1 for Unused import

Signed-off-by: John Andersen <[email protected]>

lgtm-com · 2019-08-09T08:53:31Z

This pull request introduces 2 alerts when merging 181bf51 into d4b2a35 - view on LGTM.com

new alerts:

2 for Unused import

…l: Permission denied Signed-off-by: John Andersen <[email protected]>

lgtm-com · 2019-08-10T00:29:37Z

This pull request introduces 2 alerts when merging 67c139b into d4b2a35 - view on LGTM.com

new alerts:

2 for Unused import

Signed-off-by: John Andersen <[email protected]>

…certificate chain Signed-off-by: John Andersen <[email protected]>

Signed-off-by: John Andersen <[email protected]>

commit 2a459b3 Author: Yash Lamba <[email protected]> Date: Fri Sep 6 22:18:19 2019 +0530 README: Add black badge commit 5bf5701 Author: John Andersen <[email protected]> Date: Tue Sep 3 14:18:21 2019 -0700 model: scikit: Fix CLI based configuration Signed-off-by: John Andersen <[email protected]> commit 917d997 Author: John Andersen <[email protected]> Date: Tue Sep 3 13:36:02 2019 -0700 base: Log config on init Signed-off-by: John Andersen <[email protected]> commit f4a5dda Author: John Andersen <[email protected]> Date: Tue Sep 3 08:55:04 2019 -0700 setup: Not zip safe Signed-off-by: John Andersen <[email protected]> commit 2864a11 Author: John Andersen <[email protected]> Date: Wed Aug 28 11:17:51 2019 -0700 README: Add link to master docs Signed-off-by: John Andersen <[email protected]> commit 47f76b0 Author: Yash Lamba <[email protected]> Date: Wed Aug 28 17:15:09 2019 +0530 model: scikit: Changed self to cls commit 1823b9e Author: John Andersen <[email protected]> Date: Tue Aug 27 08:48:52 2019 -0700 service: http: Fix type in securiy docs Signed-off-by: John Andersen <[email protected]> commit c4d5d2f Author: John Andersen <[email protected]> Date: Tue Aug 27 08:39:38 2019 -0700 service: http: Release HTTP service (intel#182) Signed-off-by: John Andersen <[email protected]> commit 075577a Author: Pankaj Patil <[email protected]> Date: Mon Aug 26 09:04:26 2019 +0530 examples: source: Rename testcase Fixes: intel#117 commit af19211 Author: John Andersen <[email protected]> Date: Fri Aug 23 13:44:14 2019 -0700 setup: Capitalize version and readme Signed-off-by: John Andersen <[email protected]> commit 15597c9 Author: John Andersen <[email protected]> Date: Fri Aug 23 13:36:35 2019 -0700 README: Remove link to HACKING.md Signed-off-by: John Andersen <[email protected]> commit 9fd0c55 Author: John Andersen <[email protected]> Date: Fri Aug 23 13:31:57 2019 -0700 docs: CONTRIBUTING: Merge HACKING Fixes: intel#166 Signed-off-by: John Andersen <[email protected]> commit ce03f9b Author: John Andersen <[email protected]> Date: Fri Aug 23 12:42:19 2019 -0700 cli: Enable usage via module invokation * Example: $ python3.7 -m dffml list repos Signed-off-by: John Andersen <[email protected]> commit db326cb Author: John Andersen <[email protected]> Date: Thu Aug 22 14:56:34 2019 -0700 service: dev: List entrypoints Signed-off-by: John Andersen <[email protected]> commit 7d1c6f6 Author: John Andersen <[email protected]> Date: Thu Aug 22 14:07:31 2019 -0700 model: scikit: Correct entrypoints again Signed-off-by: John Andersen <[email protected]> commit bab734e Author: Yash Lamba <[email protected]> Date: Thu Aug 22 05:36:53 2019 +0530 model: scikit: corrected entry points Fixes: intel#165 commit eb9b600 Author: John Andersen <[email protected]> Date: Fri Aug 16 13:46:51 2019 -0700 style: Correct style of all submodules * Enforce style in all submodules via ci Fixes: intel#173 Signed-off-by: John Andersen <[email protected]> commit c9cd3ea Author: John Andersen <[email protected]> Date: Fri Aug 16 11:45:04 2019 -0700 examples: shouldi: Add dev mode hack Fixes: intel#174 Signed-off-by: John Andersen <[email protected]> commit 452a182 Author: John Andersen <[email protected]> Date: Thu Aug 15 14:48:17 2019 -0700 news: origin/master August 15 2019 Signed-off-by: John Andersen <[email protected]> commit 3d04591 Author: John Andersen <[email protected]> Date: Mon Aug 12 18:34:53 2019 -0700 skel: source: entry_points correction * setup() kwargs entry_points was the service entrypoint, corrected to the source entrypoint. Fixes: intel#162 commit c854e40 Author: Sudharsana K J L <[email protected]> Date: Sun Aug 11 09:33:19 2019 +0530 source: csv: Add label Co-authored-by: John Andersen <[email protected]> Signed-off-by: John Andersen <[email protected]> commit 7afd85a Author: Yash Lamba <[email protected]> Date: Sun Aug 11 03:00:45 2019 +0530 model: scikit: Create models and configs dynamicly commit d478dee Author: Saksham Arora <[email protected]> Date: Sun Aug 11 02:32:36 2019 +0530 docs: HACKING: Added logging for testing

Signed-off-by: John Andersen <[email protected]>

lgtm-com · 2019-09-09T21:38:10Z

This pull request introduces 6 alerts when merging 39512c3 into 2a459b3 - view on LGTM.com

new alerts:

4 for Unused import
2 for Except block handles 'BaseException'

sudharsana-kjl added 6 commits July 5, 2019 15:21

add mysql_source.py

43f4897

add help for query arg in mysql_source

bd0fb5f

fix typo

fe34a34

add query format in mysql_source

a7e19fd

modify CHANGELOG

1a43843

add DictCursor in mysql source

2dbe495

johnandersen777 suggested changes Jul 9, 2019

View reviewed changes

johnandersen777 reviewed Jul 9, 2019

View reviewed changes

dffml/source/mysql_source.py Outdated Show resolved Hide resolved

sudharsana-kjl added 2 commits July 10, 2019 12:19

modify changelog

d36a2f3

modify setup.py to add aiomysql package requirement

facb3da

sudharsana-kjl added 5 commits July 10, 2019 12:33

source: mysql_source: add different args for different queries

563c04a

source: mysql_source: remove defaults for db, user, pass

f0423b5

initial test for mysql connection setup

fcb1ed4

source: mysql_source: connection verified and add test for fetch a re…

31eeffd

…po query

source: mysql: modify update()

3f4a628

johnandersen777 suggested changes Jul 15, 2019

View reviewed changes

source: mysql: modify update()

3a06cee

johnandersen777 reviewed Jul 16, 2019

View reviewed changes

dffml/source/mysql_source.py Show resolved Hide resolved

sudharsana-kjl and others added 8 commits July 16, 2019 21:43

source: mysql: update() refined

05abd5f

add db.sql file

9e39bce

source: mysql: modify convert_to_repos()

09b9bc5

source: mysql: unfold escaped string after fetching

1dd30fe

source: mysql: modify convert_to_repo to work for one repo

59b3982

Create table in mysql testcase setUP

896dd91

Signed-off-by: John Andersen <[email protected]>

source: mysql: modify help queries in args()

867dcae

add ssl for mysql source

58f4437

sudharsana-kjl added 6 commits August 1, 2019 21:04

modify test-db

efcd934

modify mysql host in test_mysql

e4d9d7f

change port in test_mysql

3897fc3

modify port in test_mysql

6a13093

remove ssl

a098103

fix style

c791b2c

johnandersen777 added this to the 0.2.3 Dot Release milestone Aug 1, 2019

Merge branch 'master' into mysql_feature

6c67cee

source: mysql: New module

181bf51

Signed-off-by: John Andersen <[email protected]>

mysql is being a fucker: line 54: /docker-entrypoint-initdb.d/dump.sq…

67c139b

…l: Permission denied Signed-off-by: John Andersen <[email protected]>

pdxjohnny and others added 14 commits August 11, 2019 17:32

source: mysql: Test in docker

b5f1acd

Signed-off-by: John Andersen <[email protected]>

merge master

2fa312a

Signed-off-by: John Andersen <[email protected]>

ci: source mysql travis

864d2ef

Signed-off-by: John Andersen <[email protected]>

source: mysql: certificate verify failed: self signed certificate in …

38b2417

…certificate chain Signed-off-by: John Andersen <[email protected]>

signing with intermedate still results in self signed error

60f837f

Signed-off-by: John Andersen <[email protected]>

internmetiate cert

cfebc4f

Signed-off-by: John Andersen <[email protected]>

mariadb and pcks8 to 1

2844a02

Signed-off-by: John Andersen <[email protected]>

tests: TLS with mariadb

f4a80c9

Signed-off-by: John Andersen <[email protected]>

merge

f7a0a58

Signed-off-by: John Andersen <[email protected]>

util: Add timing

34e2e2f

Signed-off-by: John Andersen <[email protected]>

style: Format with black

d9b08a3

Signed-off-by: John Andersen <[email protected]>

Merge remote-tracking branch 'origin/master' into mysql_feature

b57872b

fix link to readme

39512c3

Signed-off-by: John Andersen <[email protected]>

johnandersen777 merged commit c9c3f2d into intel:master Sep 10, 2019

WIP: source: Mysql Source feature #114

WIP: source: Mysql Source feature #114

Uh oh!

Conversation

sudharsana-kjl commented Jul 8, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johnandersen777 commented Jul 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

johnandersen777 commented Jul 9, 2019

Uh oh!

codecov-io commented Jul 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

johnandersen777 Jul 14, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johnandersen777 Jul 15, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sudharsana-kjl commented Jul 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lgtm-com bot commented Aug 1, 2019

Uh oh!

lgtm-com bot commented Aug 5, 2019

Uh oh!

lgtm-com bot commented Aug 9, 2019

Uh oh!

lgtm-com bot commented Aug 10, 2019

Uh oh!

lgtm-com bot commented Sep 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

johnandersen777 commented Jul 9, 2019 •

edited

Loading

codecov-io commented Jul 10, 2019 •

edited

Loading

sudharsana-kjl commented Jul 21, 2019 •

edited

Loading