Quick Start Guide¶
Installation¶
To install murphy on your machine, just install via pip:
pip install smpa-murphy
For more information on installation, check out our install guide.
Starting up Dask (optional)¶
Using Dask is optional, and while all of our code is backwards compatible with Pandas, being able to launch your own Dask Cluster or having access to the Dask Dashboard or for any of it’s other use cases
To use Dask, simply import it’s Client class and initialize with your configurations
from dask.distributed import Client
client = Client(<your configs here>)
client
You can find more information on Dask Client here
Loading the Data¶
You can load datasets by pointing to where they’ve been saved from Dozent.
The syntax for doing so is as follows:
from murphy import data_loader # -> Importing murphy
data = data_loader.DataLoader(file_find_expression = 'data/test_data/*.json.bz2') # -> You can point to another location here
twitter_dataframe = data.twitter_dataframe # -> this return a dask dataframe that is lazily computed
twitter_dataframe
This is what your output should look like (in a jupyter notebook)
You might be thinking: So, my data is going to just be loaded from the file? That’s it?
Nope! Looking at a snippet from the data_loader.DataLoader
documentation
>>> help(data_loader.DataLoader)
class DataLoader(builtins.object)
| DataLoader(file_find_expression: Union[str, pathlib.Path, List[pathlib.Path]], remove_emoji: bool = True, remove_retweets_symbols: bool = True, remove_truncated_tweets: bool = True, add_usernames: bool = True, tokenize: bool = True, filter_stopwords: bool = True, lemmatize: bool = True, language: str = 'english')
|
| Methods defined here:
|
| __init__(self, file_find_expression: Union[str, pathlib.Path, List[pathlib.Path]], remove_emoji: bool = True, remove_retweets_symbols: bool = True, remove_truncated_tweets: bool = True, add_usernames: bool = True, tokenize: bool = True, filter_stopwords: bool = True, lemmatize: bool = True, language: str = 'english')
| This is where you can specify how you want to configure the twitter dataset before you start processing it.
|
| :param file_find_expression: unix-like path that is used for listing out all of the files that we need
|
| :param remove_emoji: flag for removing emojis from all of the twitter text
|
| :param remove_retweets_symbols: flag for removing retweet strings from all of the twitter text (`RT @<retweet_username>:`)
|
| :param remove_truncated_tweets: flag for removing all tweets that are truncated, as not all information can be
| found in them
|
| :param add_usernames: flag for adding in the user names from who tweeted as a separate column instead of parsing
| it from the `user` column
|
| :param tokenize: tokenize tweets to make them easier to process
|
| :param filter_stopwords: remove stopwords from the tweets to make them easier to process
|
| :param lemmatize: lemmatize text to make it easier to process
|
| :param language: select the language that you want to work with
Here, we can see that the DataLoader class has tons of configurable parameters that we can use to make development easier, including built in tokenization, lemmatization, and more!
These are automatically run when you compute the your twitter_dataframe, meaning that these functions are automatically implemented and parallelized, right out of the box!
Now what?¶
Now, you can explore the data to your heart’s content! We suggest looking over this Dask Tutorial if you’re not familiar with Dask already, as it’ll make exploring the dataset easier