You\u2019ve probably heard of Kaggle data science competitions, but did you know that Kaggle has many other features that can help you with your next machine learning project? For people looking for datasets for their next machine learning project, Kaggle allows you to access public datasets by others and share your own datasets. For those looking to build and train their own machine learning models, Kaggle also offers an in-browser notebook environment and some free GPU hours. You can also look at other people\u2019s public notebooks as well!<\/p>\n
Other than the website, Kaggle also has a command-line interface (CLI) which you can use within the command line to access and download datasets.<\/p>\n
Let\u2019s dive right in and explore what Kaggle has to offer!<\/p>\n
After completing this tutorial, you will learn:<\/p>\n
- \n
- What is Kaggle?<\/li>\n
- How you can use Kaggle as part of your machine learning pipeline<\/li>\n
- Using Kaggle API\u2019s Command Line Interface (CLI)<\/li>\n<\/ul>\n
Let\u2019s get started!<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/stefan-widua-kOuaZs7jDZE-unsplash-scaled.jpg\")
<\/p>\nUsing Kaggle in Machine Learning Projects
Photo by Stefan Widua<\/a>. Some rights reserved.<\/p>\n<\/div>\nOverview<\/h2>\n
This tutorial is split into five parts; they are:<\/p>\n
- \n
- What is Kaggle?<\/li>\n
- Setting up Kaggle Notebooks<\/li>\n
- Using Kaggle Notebooks with GPUs\/TPUs<\/li>\n
- Using Kaggle Datasets with Kaggle Notebooks<\/li>\n
- Using Kaggle Datasets with Kaggle CLI tool<\/li>\n<\/ul>\n
What Is Kaggle?<\/h2>\n
Kaggle is probably most well known for the data science competitions that it hosts, with some of them offering 5-figure prize pools and seeing hundreds of teams participating. Besides these competitions, Kaggle also allows users to publish and search for datasets, which they can use for their machine learning projects. To use these datasets, you can use Kaggle notebooks within your browser or Kaggle\u2019s public API to download their datasets which you can then use for your machine learning projects.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/kaggle_featured_competitions.png\")
<\/a><\/p>\nKaggle Competitions<\/p>\n<\/div>\n
In addition to that, Kaggle also offers some courses and a discussions page for you to learn more about machine learning and talk with other machine learning practitioners!<\/p>\n
For the rest of this article, we\u2019ll focus on how we can use Kaggle\u2019s datasets and notebooks to help us when working on our own machine learning projects or finding new projects to work on.<\/p>\n
Setting up Kaggle Notebooks<\/h2>\n
To get started with Kaggle Notebooks, you\u2019ll need to create a Kaggle account either using an existing Google account or creating one using your email.<\/p>\n
Then, go to the \u201cCode\u201d page.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/kaggle_sidebar_notebook.png\")
<\/a><\/p>\nLeft Sidebar of Kaggle Home Page, Code Tab<\/p>\n<\/div>\n
You will then be able to see your own notebooks as well as public notebooks by others. To create your own notebook, click on New Notebook.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/kaggle_code_page.png\")
<\/a><\/p>\nKaggle Code Page<\/p>\n<\/div>\n
This will create your new notebook, which looks like a Jupyter notebook, with many similar commands and shortcuts.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/kaggle_empty_notebook.png\")
<\/a><\/p>\nKaggle Notebook<\/p>\n<\/div>\n
You can also toggle between a notebook editor and script editor by going to File -> Editor Type.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/kaggle_toggle_script.png\")
<\/a><\/p>\nChanging Editor Type in Kaggle Notebook<\/p>\n<\/div>\n
Changing the editor type to script shows this instead:<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/kaggle_empty_script.png\")
<\/a><\/p>\nKaggle Notebook Script Editor Type<\/p>\n<\/div>\n
Using Kaggle with GPUs\/TPUs<\/h2>\n
Who doesn\u2019t love free GPU time for machine learning projects? GPUs can help to massively speed up the training and inference of machine learning models, especially with deep learning models.<\/p>\n
Kaggle comes with some free allocation of GPUs and TPUs, which you can use for your projects. At the time of this writing, the availability is 30 hours a week for GPUs and 20 hours a week for TPUs after verifying your account with a phone number.<\/p>\n
To attach an accelerator to your notebook, go to Settings \u25b7 Environment \u25b7 Preferences.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/kaggle_environment_preferences.png\")
<\/a><\/p>\nChanging Kaggle Notebook Environment preferences<\/p>\n<\/div>\n
You\u2019ll be asked to verify your account with a phone number.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/verify_phone.png\")
<\/a><\/p>\nVerify phone number<\/p>\n<\/div>\n
And then presented with this page which lists the amount of availability you have left and mentions that turning on GPUs will reduce the number of CPUs available, so it\u2019s probably only a good idea when doing training\/inference with neural networks.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/use_accelerator.png\")
<\/a><\/p>\nAdding GPU Accelerator to Kaggle Notebook<\/p>\n<\/div>\n
Using Kaggle Datasets with Kaggle Notebooks<\/h2>\n
Machine learning projects are data-hungry monsters, and finding datasets for our current projects or looking for datasets to start new projects is always a chore. Luckily, Kaggle has a rich collection of datasets contributed by users and from competitions. These datasets can be a treasure trove for people looking for data for their current machine learning project or people looking for new ideas for projects.<\/p>\n
Let\u2019s explore how we can add these datasets to our Kaggle notebook.<\/p>\n
First, click on Add data on the right sidebar.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/add_data.png\")
<\/a><\/p>\nAdding Datasets to Kaggle Notebook Environment<\/p>\n<\/div>\n
A window should appear that shows you some of the publicly available datasets and gives you the option to upload your own dataset for use with your Kaggle notebook.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/kaggle_datasets.png\")
<\/a><\/p>\nSearching Through Kaggle datasets<\/p>\n<\/div>\n
I\u2019ll be using the classic titanic dataset as my example for this tutorial, which you can find by keying your search terms into the search bar on the top right of the window.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/search_titanic_dataset.png\")
<\/a><\/p>\nKaggle Datasets Filtered with \u201cTitanic\u201d Keyword<\/p>\n<\/div>\n
After that, the dataset is available to be used by the notebook. To access the files, take a look at the path for the file and prepend
..\/input\/{path}<\/code>. For example, the file path for the titanic dataset is:<\/p>\n..\/input\/titanic\/train_and_test2.csv<\/pre>\n
In the notebook, we can read the data using:<\/p>\n
import pandas\r\n\r\npandas.read_csv(\"..\/input\/titanic\/train_and_test2.csv\")<\/pre>\n
This gets us the data from the file:<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/kaggle_notebook_read_dataset.png\")
<\/a><\/p>\nUsing Titanic Dataset in Kaggle Notebook<\/p>\n<\/div>\n
Using Kaggle Datasets with Kaggle CLI Tool<\/h2>\n
Kaggle also has a public API with a CLI tool which we can use to download datasets, interact with competitions, and much more. We\u2019ll be looking at how to set up and download Kaggle datasets using the CLI tool.<\/p>\n
To get started, install the CLI tool using:<\/p>\n
pip install kaggle<\/pre>\n
For Mac\/Linux users, you might need:<\/p>\n
pip install --user kaggle<\/pre>\n
Then, you\u2019ll need to create an API token for authentication. Go to Kaggle\u2019s webpage, click on your profile icon in the top right corner and go to Account.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/kaggle_account.png\")
<\/a><\/p>\nGoing to Kaggle Account Settings<\/p>\n<\/div>\n
From there, scroll down to Create New API Token:<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/create_api_token.png\")
<\/a><\/p>\nGenerating New API Token for Kaggle Public API<\/p>\n<\/div>\n
This will download a
kaggle.json<\/code> file that you\u2019ll use to authenticate yourself with the Kaggle CLI tool. You will have to place it in the correct location for it to work. For Linux\/Mac\/Unix-based operating systems, this should be placed at~\/.kaggle\/kaggle.json<\/code>, and for Windows users, it should be placed atC:Users<Windows-username>.kagglekaggle.json<\/code>. Placing it in the wrong location and callingkaggle<\/code> in the command line will give an error:<\/p>\nOSError: Could not find kaggle.json. Make sure it\u2019s location in \u2026 Or use the environment method<\/pre>\n
Now, let\u2019s get started on downloading those datasets!<\/p>\n
To search for datasets using a search term, e.g., titanic, we can use:<\/p>\n
kaggle datasets list -s titanic<\/pre>\n
Searching for titanic, we get:<\/p>\n
$ kaggle datasets list -s titanic\r\nref title size lastUpdated downloadCount voteCount usabilityRating\r\n----------------------------------------------------------- --------------------------------------------- ----- ------------------- ------------- --------- ---------------\r\ndatasets\/heptapod\/titanic Titanic 11KB 2017-05-16 08:14:22 37681 739 0.7058824\r\ndatasets\/azeembootwala\/titanic Titanic 12KB 2017-06-05 12:14:37 13104 145 0.8235294\r\ndatasets\/brendan45774\/test-file Titanic dataset 11KB 2021-12-02 16:11:42 19348 251 1.0\r\ndatasets\/rahulsah06\/titanic Titanic 34KB 2019-09-16 14:43:23 3619 43 0.6764706\r\ndatasets\/prkukunoor\/TitanicDataset Titanic 135KB 2017-01-03 22:01:13 4719 24 0.5882353\r\ndatasets\/hesh97\/titanicdataset-traincsv Titanic-Dataset (train.csv) 22KB 2018-02-02 04:51:06 54111 377 0.4117647\r\ndatasets\/fossouodonald\/titaniccsv Titanic csv 1KB 2016-11-07 09:44:58 8615 50 0.5882353\r\ndatasets\/broaniki\/titanic titanic 717KB 2018-01-30 04:08:45 8004 128 0.1764706\r\ndatasets\/pavlofesenko\/titanic-extended Titanic extended dataset (Kaggle + Wikipedia) 134KB 2019-03-06 09:53:24 8779 130 0.9411765\r\ndatasets\/jamesleslie\/titanic-cleaned-data Titanic: cleaned data 36KB 2018-11-21 11:50:18 4846 53 0.7647059\r\ndatasets\/kittisaks\/testtitanic test titanic 22KB 2017-03-13 15:13:12 1658 32 0.64705884\r\ndatasets\/yasserh\/titanic-dataset Titanic Dataset 22KB 2021-12-24 14:53:06 1011 25 1.0\r\ndatasets\/abhinavralhan\/titanic titanic 22KB 2017-07-30 11:07:55 628 11 0.8235294\r\ndatasets\/cities\/titanic123 Titanic Dataset Analysis 22KB 2017-02-07 23:15:54 1585 29 0.5294118\r\ndatasets\/brendan45774\/gender-submisson Titanic: all ones csv file 942B 2021-02-12 19:18:32 459 34 0.9411765\r\ndatasets\/harunshimanto\/titanic-solution-for-beginners-guide Titanic Solution for Beginner's Guide 34KB 2018-03-12 17:47:06 1444 21 0.7058824\r\ndatasets\/ibrahimelsayed182\/titanic-dataset Titanic dataset 6KB 2022-01-27 07:41:54 334 8 1.0\r\ndatasets\/sureshbhusare\/titanic-dataset-from-kaggle Titanic DataSet from Kaggle 33KB 2017-10-12 04:49:39 2688 27 0.4117647\r\ndatasets\/shuofxz\/titanic-machine-learning-from-disaster Titanic: Machine Learning from Disaster 33KB 2017-10-15 10:05:34 3867 55 0.29411766\r\ndatasets\/vinicius150987\/titanic3 The Complete Titanic Dataset 277KB 2020-01-04 18:24:11 1459 23 0.64705884<\/pre>\n
To download the first dataset in that list, we can use:<\/p>\n
kaggle datasets download -d heptapod\/titanic --unzip<\/pre>\n
Using a Jupyter notebook to read the file, similar to the Kaggle notebook example, gives us:<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/04\/jupyter_titanic.png\")
<\/a><\/p>\nUsing Titanic Dataset in Jupyter Notebook<\/p>\n<\/div>\n
Of course, some datasets are so large in size that you may not want to keep them on your own disk. Nonetheless, this is one of the free resources provided by Kaggle for your machine learning projects!<\/p>\n
Further Reading<\/h2>\n
This section provides more resources if you\u2019re interested in going deeper into the topic.<\/p>\n
- \n
- Kaggle: https:\/\/www.kaggle.com<\/a>\n<\/li>\n
- Kaggle API documentation: https:\/\/www.kaggle.com\/docs\/api<\/a>\n<\/li>\n<\/ul>\n
Summary<\/h2>\n
In this tutorial, you learned what Kaggle is , how we can use Kaggle to get datasets, and even for some free GPU\/TPU instances within Kaggle Notebooks. You\u2019ve also seen how we can use Kaggle API\u2019s CLI tool to download datasets for us to use in our local environments.<\/p>\n
Specifically, you learnt:<\/p>\n
- \n
- What is Kaggle<\/li>\n
- How to use Kaggle notebooks along with their GPU\/TPU accelerator<\/li>\n
- How to use Kaggle datasets in Kaggle notebooks or download them using Kaggle\u2019s CLI tool<\/li>\n<\/ul>\n
The post Using Kaggle in Machine Learning Projects<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n
Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
- Kaggle API documentation: https:\/\/www.kaggle.com\/docs\/api<\/a>\n<\/li>\n<\/ul>\n
- Kaggle: https:\/\/www.kaggle.com<\/a>\n<\/li>\n