replaceWhere

If we want to replace content of table or file in a path below can be possible.

• The replaceWhere option atomically replaces all records that match a given predicate.

• You can replace directories of data based on how tables are partitioned using dynamic partition overwrites.

Python:

replace_data.write
  .mode("overwrite")
  .option("replaceWhere", "start_date >= '2017-01-02' AND end_date <= '2017-01-30'")
  .save("/tmp1/delta/events")

SQL:

INSERT INTO TABLE events REPLACE WHERE start_data >= '2017-01-01' AND end_date <= '2017-01-31' SELECT * FROM replace_data

Magic commands

 

Following are possible magic commands available in Databricks

1. %python
2. %sql
3. %scala
4. %sh
5. %fs → Alternatively, one can use dbutils.fs
6. %md

Note:

• The first line in the cell must be the magic command
• One cell allows only one magic command
• Magic commands are case sensitive
• When you change the default language, all cells in that notebook automatically add a magic command of the previous default language.