Regex:Finding Patterns in Strings

Problem

You need to determine whether a String contains a regular expression pattern.

Solution

Create a Regex object by invoking the .r method on a String, and then use that pattern with findFirstIn when you’re looking for one match, and findAllIn when looking for all matches.

To demonstrate this, first create a Regex for the pattern you want to search for, in this case, a sequence of one or more numeric characters:

scala> val numPattern = "[0-9]+".r
numPattern: scala.util.matching.Regex = [0-9]+

Next, create a sample String you can search:

scala> val address = "123 Main Street Suite 101"
address: java.lang.String = 123 Main Street Suite 101

The findFirstIn method finds the first match:

scala> val match1 = numPattern.findFirstIn(address)
match1: Option[String] = Some(123)

(Notice that this method returns an Option[String]. I’ll dig into that in the Discussion.)

When looking for multiple matches, use the findAllIn method:

scala> val matches = numPattern.findAllIn(address)
matches: scala.util.matching.Regex.MatchIterator = non-empty iterator

As you can see, findAllIn returns an iterator, which lets you loop over the results:

scala> matches.foreach(println)
123
101

If findAllIn doesn’t find any results, an empty iterator is returned, so you can still write your code just like that—you don’t need to check to see if the result isnull. If you’d rather have the results as an Array, add the toArray method after the findAllIn call:

scala> val matches = numPattern.findAllIn(address).toArray
matches: Array[String] = Array(123, 101)

If there are no matches, this approach yields an empty Array. Other methods like toList, toSeq, and toVector are also available.

Discussion

Using the .r method on a String is the easiest way to create a Regex object. Another approach is to import the Regex class, create a Regex instance, and then use the instance in the same way:

scala> import scala.util.matching.Regex
import scala.util.matching.Regex

scala> val numPattern = new Regex("[0-9]+")
numPattern: scala.util.matching.Regex = [0-9]+

scala> val address = "123 Main Street Suite 101"
address: java.lang.String = 123 Main Street Suite 101

scala> val match1 = numPattern.findFirstIn(address)
match1: Option[String] = Some(123)

Although this is a bit more work, it’s also more obvious. I’ve found that it can be easy to overlook the .r at the end of a String (and then spend a few minutes wondering how the code I saw could possibly work).

Handling the Option returned by findFirstIn

As mentioned in the Solution, the findFirstIn method finds the first match in the String and returns an Option[String]:

scala> val match1 = numPattern.findFirstIn(address)
match1: Option[String] = Some(123)

The Option/Some/None pattern is discussed in detail in Recipe 20.6, but the simple way to think about an Option is that it’s a container that holds either zero or one values. In the case of findFirstIn, if it succeeds, it returns the string “123” as a Some(123), as shown in this example. However, if it fails to find the pattern in the string it’s searching, it will return a None, as shown here:

scala> val address = "No address given"
address: String = No address given

scala> val match1 = numPattern.findFirstIn(address)
match1: Option[String] = None

To summarize, a method defined to return an Option[String] will either return a Some(String), or a None.

The normal way to work with an Option is to use one of these approaches:

• Call getOrElse on the value.
• Use the Option in a match expression.
• Use the Option in a foreach loop.

Recipe 20.6 describes those approaches in detail, but they’re demonstrated here for your convenience.

With the getOrElse approach, you attempt to “get” the result, while also specifying a default value that should be used if the method failed:

scala> val result = numPattern.findFirstIn(address).getOrElse("no match")
result: String = 123

Because an Option is a collection of zero or one elements, an experienced Scala developer will also use a foreach loop in this situation:

numPattern.findFirstIn(address).foreach { e =>
  // perform the next step in your algorithm,
  // operating on the value 'e'
}

A match expression also provides a very readable solution to the problem:

match1 match {
  case Some(s) => println(s"Found: $s")
  case None =>
}

See Recipe 20.6 for more information.

To summarize this approach, the following REPL example shows the complete process of creating a Regex, searching a String with findFirstIn, and then using a foreach loop on the resulting match:

scala> val numPattern = "[0-9]+".r
numPattern: scala.util.matching.Regex = [0-9]+

scala> val address = "123 Main Street Suite 101"
address: String = 123 Main Street Suite 101

scala> val match1 = numPattern.findFirstIn(address)
match1: Option[String] = Some(123)

scala> match1.foreach { e =>
     |   println(s"Found a match: $e")
     | }
Found a match: 123

Importance of ? in Regex

It is the difference between greedy and non-greedy quantifiers.

Consider the input 101000000000100.

Using 1.*1, * is greedy - it will match all the way to the end, and then backtrack until it can match 1, leaving you with 1010000000001.
.*? is non-greedy. * will match nothing, but then will try to match extra characters until it matches 1, eventually matching 101.

All quantifiers have a non-greedy mode: .*?, .+?, .{2,6}?, and even .??.

In your case, a similar pattern could be <([^>]*)> - matching anything but a greater-than sign (strictly speaking, it matches zero or more characters other than > in-between < and >).

Reference:https://stackoverflow.com/questions/3075130/what-is-the-difference-between-and-regular-expressions

Find the text between two characters using Regex

scala> import scala.util.matching.Regex
import scala.util.matching.Regex

scala> val keyValPattern: Regex = "(?<=\\().*?(?=\\))".r
keyValPattern: scala.util.matching.Regex = (?<=\().*?(?=\))

scala> val input: String ="heelo(geloo))lodk"
input: String = heelo(geloo))lodk

scala> println(keyValPattern findFirstIn input)
Some(geloo)

scala> val keyValPattern: Regex = "(?<=\\().*?(?<=\\))".r
keyValPattern: scala.util.matching.Regex = (?<=\().*?(?<=\))

scala> println(keyValPattern findFirstIn input)
Some(geloo))

Note: Here in above case (?<='character') represents a group. we used \\( or \\) as starting and ending characters. .*? represents all the text between 2 groups. 

StructType and StructFields

StructType — Data Type for Schema Definition

StructType is a built-in data type that is a collection of StructFields.

StructType is used to define a schema or its part.

You can compare two StructType instances to see whether they are equal.

import org.apache.spark.sql.types.StructType

val schemaUntyped = new StructType()
  .add("a", "int")
  .add("b", "string")

import org.apache.spark.sql.types.{IntegerType, StringType}
val schemaTyped = new StructType()
  .add("a", IntegerType)
  .add("b", StringType)

scala> schemaUntyped == schemaTyped
res0: Boolean = true

Hive Partitioning

//create a stage table with out partition.
hive> drop table emp_det_stage;
OK
Time taken: 0.079 seconds

hive> create table emp_det_stage(name string,dept string,exp int, loc string) row format delimited fields terminated by ',';
OK
Time taken: 0.088 seconds

hive> Load data local Inpath "/home/hadoop/Partition.csv" overwrite into table emp_det_stage;
Loading data to table default.emp_det_stage
OK
Time taken: 0.42 seconds
//view the loaded data
hive> select * from emp_det_stage;
OK
emp_det_stage.name      emp_det_stage.dept      emp_det_stage.exp       emp_det_stage.loc
Kartheek        BI      5       Hyd
Raj     Apps    5       Mas
mahesh  BI      5       Hyd
Denesh  BI      6       Hyd
Rajesh  Frontend        7       KOL
Time taken: 0.125 seconds, Fetched: 5 row(s)

//create an actual table with static partition:

hive> create table emp_det_part(name string,dept string,exp int) partitioned by (loc string);

OK

Time taken: 0.067 seconds

hive>  insert overwrite table emp_det_part partition(loc='Hyd') select name,dept,exp from emp_det_stage where loc='Hyd';

//verify data

hive> dfs -ls /user/hive/warehouse/emp_det_part/loc=Hyd;

Found 1 items

-rwxrwxrwt   1 hadoop hadoop         38 2019-07-08 12:08 /user/hive/warehouse/emp_det_part/loc=Hyd/000000_0

//create an actual table with dynamic partition:

hive> set hive.exec.dynamic.partition.mode=nonstrict;

hive> insert overwrite table emp_det_part partition(loc) select * from emp_det_stage;

//verify data files

hive> dfs -ls /user/hive/warehouse/emp_det_part/

    > ;

Found 3 items

drwxrwxrwt   - hadoop hadoop          0 2019-07-08 12:10 /user/hive/warehouse/emp_det_part/loc=Hyd

drwxrwxrwt   - hadoop hadoop          0 2019-07-08 12:10 /user/hive/warehouse/emp_det_part/loc=KOL

drwxrwxrwt   - hadoop hadoop          0 2019-07-08 12:10 /user/hive/warehouse/emp_det_part/loc=Mas

Functions in Hive

hive> desc T_UNSTRUCTURE;
OK
col_name        data_type       comment
emp_id                  int
name                    map<string,string>
addr                    struct<City:string,Pin:int>
skill_set               array<string>
Time taken: 0.027 seconds, Fetched: 4 row(s)


hive> select size(skill_set),array_contains(skill_set,'Hadoop'),sort_array(skill_set),concat_ws("$",skill_set) from T_UNSTRUCTURE;
OK
_c0     _c1     _c2     _c3
2       false   ["'Hadoop'","'OBIEE'"]  'Hadoop'$'OBIEE'
2       false   ["'Chocolate'","'oracle'"]      'oracle'$'Chocolate'
Time taken: 0.201 seconds, Fetched: 2 row(s)

Explode

Explode: it's a UDTF which can be used outside the select statement with "Lateral" keyword for flattening the collection objects.

hive> desc T_UNSTRUCTURE;
OK
col_name        data_type       comment
emp_id                  int
name                    map<string,string>
addr                    struct<City:string,Pin:int>
skill_set               array<string>
Time taken: 0.027 seconds, Fetched: 4 row(s)


hive> select * from T_UNSTRUCTURE;
OK
t_unstructure.emp_id    t_unstructure.name      t_unstructure.addr      t_unstructure.skill_set
10      {"first":"Amit","Last":"Mishra"}        {"city":"Blr","pin":1}  ["'Hadoop'","'OBIEE'"]
20      {"first":"Ramesh","Last":"Nayak"}       {"city":"Mas","pin":2}  ["'oracle'","'Chocolate'"]
Time taken: 0.32 seconds, Fetched: 2 row(s)


hive> select emp_id,skill from T_UNSTRUCTURE Lateral view explode(skill_set) temp_table as skill;
OK
emp_id  skill
10      'Hadoop'
10      'OBIEE'
20      'oracle'
20      'Chocolate'
Time taken: 0.104 seconds, Fetched: 4 row(s)

Hive configuration to show table header

set hive.cli.print.header=True;

Create table with Collections columns

 sample Unstructure.csv

10 first:Amit,Last:Mishra Blr,1 'Hadoop','OBIEE'
20 first:Ramesh,Last:Nayak Mas,2 'oracle','Chocolate'

move the above to the local file system.

create a table in Hive shell.

hive> create table T_UNSTRUCTURE (emp_id int,name map<string,string>,addr struct<City:String,Pin:int>,skill_set array<string>) row format delimited fields terminated by '\t' collection items terminated by ',' map keys terminated by ':';

Load data into the table with the below statement.

LOAD DATA LOCAL INPATH '/home/hadoop/Unstructure.csv' OVERWRITE INTO table T_UNSTRUCTURE;

Look at the description of the table

hive> desc T_UNSTRUCTURE;

OK

col_name        data_type       comment

emp_id                  int

name                    map<string,string>

addr                    struct<City:string,Pin:int>

skill_set               array<string>

Access the elements of table.

hive> select * from T_UNSTRUCTURE;

OK

t_unstructure.emp_id    t_unstructure.name      t_unstructure.addr      t_unstructure.skill_set

10      {"first":"Amit","Last":"Mishra"}        {"city":"Blr","pin":1}  ["'Hadoop'","'OBIEE'"]

20      {"first":"Ramesh","Last":"Nayak"}       {"city":"Mas","pin":2}  ["'oracle'","'Chocolate'"]

Time taken: 0.216 seconds, Fetched: 2 row(s)