Skip to content

Latest commit

 

History

History
118 lines (73 loc) · 5.6 KB

get-data-onelake.md

File metadata and controls

118 lines (73 loc) · 5.6 KB
title description ms.reviewer ms.author author ms.topic ms.custom ms.date ms.search.form
Get data from OneLake
Learn how to get data from OneLake into a KQL database in Real-Time Intelligence.
tzgitlin
yaschust
YaelSchuster
how-to
build-2023
ignite-2023
04/21/2024
Get data in a KQL Database

Get data from OneLake

In this article, you learn how to get data from OneLake into either a new or existing table.

Prerequisites

Copy file path from Lakehouse

  1. In the experience switcher, choose Data Engineering.

  2. Select the Lakehouse environment containing the data source you want to use.

  3. Place your cursor over the desired file and select the More menu, then select Properties.

    [!IMPORTANT]

    • Folder paths aren't supported.
    • Wildcards (*) aren't supported.

    :::image type="content" source="media/get-data-onelake/lakehouse-file-menu.png" alt-text="Screenshot of a Lakehouse file's dropdown menu. The option titled Properties is highlighted." lightbox="media/get-data-onelake/lakehouse-file-menu.png":::

  4. Under URL, select the Copy to clipboard icon and save it somewhere to retrieve in a later step.

    :::image type="content" source="media/get-data-onelake/lakehouse-file-properties.png" alt-text="Screenshot of a Lakehouse file's Properties pane. The copy icon to the right of the file's URL is highlighted." lightbox="media/get-data-onelake/lakehouse-file-properties.png":::

  5. Return to your workspace and select a KQL database.

Source

  1. On the lower ribbon of your KQL database, select Get Data.

    In the Get data window, the Source tab is selected.

  2. Select the data source from the available list. In this example, you're ingesting data from OneLake.

    [!INCLUDE get-data-kql]

Configure

  1. Select a target table. If you want to ingest data into a new table, select +New table and enter a table name.

    [!NOTE] Table names can be up to 1024 characters including spaces, alphanumeric, hyphens, and underscores. Special characters aren't supported.

  2. In OneLake file, paste the file path of the Lakehouse you copied in Copy file path from Lakehouse.

    [!NOTE] You can add up to 10 items of up to 1-GB uncompressed size each.

    :::image type="content" source="media/get-data-onelake/configure-tab.png" alt-text="Screenshot of configure tab with new table entered and a OneLake file path added." lightbox="media/get-data-onelake/configure-tab.png":::

  3. Select Next.

Inspect

The Inspect tab opens with a preview of the data.

To complete the ingestion process, select Finish.

:::image type="content" source="media/get-data-onelake/inspect-data.png" alt-text="Screenshot of the inspect tab." lightbox="media/get-data-onelake/inspect-data.png":::

Optionally:

[!INCLUDE get-data-edit-columns]

:::image type="content" source="media/get-data-onelake/edit-columns.png" alt-text="Screenshot of columns open for editing." lightbox="media/get-data-onelake/edit-columns.png":::

[!INCLUDE mapping-transformations]

Advanced options based on data type

Tabular (CSV, TSV, PSV):

  • If you're ingesting tabular formats in an existing table, you can select Advanced > Keep table schema. Tabular data doesn't necessarily include the column names that are used to map source data to the existing columns. When this option is checked, mapping is done by-order, and the table schema remains the same. If this option is unchecked, new columns are created for incoming data, regardless of data structure.

  • To use the first row as column names, select Advanced > First row is column header.

    :::image type="content" source="media/get-data-onelake/advanced-csv.png" alt-text="Screenshot of advanced CSV options.":::

JSON:

  • To determine column division of JSON data, select Advanced > Nested levels, from 1 to 100.

  • If you select Advanced > Skip JSON lines with errors, the data is ingested in JSON format. If you leave this check box unselected, the data is ingested in multijson format.

    :::image type="content" source="media/get-data-onelake/advanced-json.png" alt-text="Screenshot of advanced JSON options.":::

Summary

In the Data preparation window, all three steps are marked with green check marks when data ingestion finishes successfully. You can select a card to query, drop the ingested data, or see a dashboard of your ingestion summary.

:::image type="content" source="media/get-data-onelake/summary.png" alt-text="Screenshot of summary page with successful ingestion completed." lightbox="media/get-data-onelake/summary.png":::

Related content