Dev Stack Source

Answer : Convert the DataFrame's content (e.g. Lat and Lon columns) into appropriate Shapely geometries first and then use them together with the original DataFrame to create a GeoDataFrame. from geopandas import GeoDataFrame from shapely.geometry import Point geometry = [Point(xy) for xy in zip(df.Lon, df.Lat)] df = df.drop(['Lon', 'Lat'], axis=1) gdf = GeoDataFrame(df, crs="EPSG:4326", geometry=geometry) Result: Date/Time ID geometry 0 4/1/2014 0:11:00 140 POINT (-73.95489999999999 40.769) 1 4/1/2014 0:17:00 NaN POINT (-74.03449999999999 40.7267) Since the geometries often come in the WKT format, I thought I'd include an example for that case as well: import geopandas as gpd import shapely.wkt geometry = df['wktcolumn'].map(shapely.wkt.loads) df = df.drop('wktcolumn', axis=1) gdf = gpd.GeoDataFrame(df, crs="EPSG:4326", geometry=geometry) Update 201912: The official documentation at h...

Answer : hashlib.md5 takes a single string as input -- you can't pass it an array of values as you can with some NumPy/Pandas functions. So instead, you could use a list comprehension to build a list of md5sums: ob['md5'] = [hashlib.md5(val).hexdigest() for val in ob['ssno']] In case you are hashing to SHA256, you'll need to encode your string first to (probably) UTF-8: ob['sha256'] = [hashlib.sha256(val.encode('UTF-8')).hexdigest() for val in ob['ssno']]

Answer : You can't: DataFrame columns are Series , by definition. That said, if you make the dtype (the type of all the elements) datetime-like, then you can access the quantities you want via the .dt accessor (docs): >>> df["TimeReviewed"] = pd.to_datetime(df["TimeReviewed"]) >>> df["TimeReviewed"] 205 76032930 2015-01-24 00:05:27.513000 232 76032930 2015-01-24 00:06:46.703000 233 76032930 2015-01-24 00:06:56.707000 413 76032930 2015-01-24 00:14:24.957000 565 76032930 2015-01-24 00:23:07.220000 Name: TimeReviewed, dtype: datetime64[ns] >>> df["TimeReviewed"].dt <pandas.tseries.common.DatetimeProperties object at 0xb10da60c> >>> df["TimeReviewed"].dt.year 205 76032930 2015 232 76032930 2015 233 76032930 2015 413 76032930 2015 565 76032930 2015 dtype: int64 >>> df["TimeReviewed"].dt.month 205 76032930 1 232 76032930 1 233 7...

Answer : The to_dict() method sets the column names as dictionary keys so you'll need to reshape your DataFrame slightly. Setting the 'ID' column as the index and then transposing the DataFrame is one way to achieve this. to_dict() also accepts an 'orient' argument which you'll need in order to output a list of values for each column. Otherwise, a dictionary of the form {index: value} will be returned for each column. These steps can be done with the following line: >>> df.set_index('ID').T.to_dict('list') {'p': [1, 3, 2], 'q': [4, 3, 2], 'r': [4, 0, 9]} In case a different dictionary format is needed, here are examples of the possible orient arguments. Consider the following simple DataFrame: >>> df = pd.DataFrame({'a': ['red', 'yellow', 'blue'], 'b': [0.5, 0.25, 0.125]}) >>> df a b 0 red 0.500 1 yellow 0.250 2 blue 0.125 The...

Answer : You can use datetime methods. from datetime import datetime a = '20160228' date = datetime.strptime(a, '%Y%m%d').strftime('%m/%d/%Y') Good Luck; Build a new column with applymap : import pandas as pd dates = [ 20160228, 20161231, 20160618, 20170123, 20151124, ] df = pd.DataFrame(data=list(enumerate(dates, start=1)), columns=['id','int_date']) df[['str_date']] = df[['int_date']].applymap(str).applymap(lambda s: "{}/{}/{}".format(s[4:6],s[6:], s[0:4])) print(df) Emits: $ python test.py id int_date str_date 0 1 20160228 02/28/2016 1 2 20161231 12/31/2016 2 3 20160618 06/18/2016 3 4 20170123 01/23/2017 4 5 20151124 11/24/2015 There is bound to be a better solution to this, but since you have zeroes instead of single-digit elements in your date (i.e. 06 instead of 6), why not just convert it to string and convert the subsections? using datetime would also get you the...

Answer : Use groupby and agg , and aggregate only unique values by calling Series.unique : df.astype(str).groupby('prop1').agg(lambda x: ','.join(x.unique())) prop2 prop3 prop4 prop1 K20 12,1,66 travis,leo 10.0,4.0 L30 3,54,11,10 bob,john 11.2,10.0 df.astype(str).groupby('prop1', sort=False).agg(lambda x: ','.join(x.unique())) prop2 prop3 prop4 prop1 L30 3,54,11,10 bob,john 11.2,10.0 K20 12,1,66 travis,leo 10.0,4.0 If handling NaNs is important, call fillna in advance: import re df.fillna('').astype(str).groupby('prop1').agg( lambda x: re.sub(',+', ',', ','.join(x.unique())) ) prop2 prop3 prop4 prop1 K20 12,1,66 travis,leo 10.0,4.0 L30 3,54,11,10 bob,john 11.2,10.0

Answer : With newer Pandas versions, if someone is interested, here a slightly different solution with Pandas only: ax = data.plot(kind='hist',subplots=True,sharex=True,sharey=True,title='My title') You can use suptitle() : import pylab as pl from pandas import * data = DataFrame(np.random.randn(500).reshape(100,5), columns=list('abcde')) axes = data.hist(sharey=True, sharex=True) pl.suptitle("This is Figure title") I found a better way: plt.subplot(2,3,1) # if use subplot df = pd.read_csv('documents',low_memory=False) df['column'].hist() plt.title('your title') It is very easy, display well at the top, and will not mess up your subplot.

Answer : You can use the to_pydatetime method to be more explicit: In [11]: ts = pd.Timestamp('2014-01-23 00:00:00', tz=None) In [12]: ts.to_pydatetime() Out[12]: datetime.datetime(2014, 1, 23, 0, 0) It's also available on a DatetimeIndex: In [13]: rng = pd.date_range('1/10/2011', periods=3, freq='D') In [14]: rng.to_pydatetime() Out[14]: array([datetime.datetime(2011, 1, 10, 0, 0), datetime.datetime(2011, 1, 11, 0, 0), datetime.datetime(2011, 1, 12, 0, 0)], dtype=object) Pandas Timestamp to datetime.datetime: pd.Timestamp('2014-01-23 00:00:00', tz=None).to_pydatetime() datetime.datetime to Timestamp pd.Timestamp(datetime(2014, 1, 23)) >>> pd.Timestamp('2014-01-23 00:00:00', tz=None).to_datetime() datetime.datetime(2014, 1, 23, 0, 0) >>> pd.Timestamp(datetime.date(2014, 3, 26)) Timestamp('2014-03-26 00:00:00')

Answer : Pandas timestamp differences returns a datetime.timedelta object. This can easily be converted into hours by using the *as_type* method, like so import pandas df = pandas.DataFrame(columns=['to','fr','ans']) df.to = [pandas.Timestamp('2014-01-24 13:03:12.050000'), pandas.Timestamp('2014-01-27 11:57:18.240000'), pandas.Timestamp('2014-01-23 10:07:47.660000')] df.fr = [pandas.Timestamp('2014-01-26 23:41:21.870000'), pandas.Timestamp('2014-01-27 15:38:22.540000'), pandas.Timestamp('2014-01-23 18:50:41.420000')] (df.fr-df.to).astype('timedelta64[h]') to yield, 0 58 1 3 2 8 dtype: float64 This was driving me bonkers as the .astype() solution above didn't work for me. But I found another way. Haven't timed it or anything, but might work for others out there: t1 = pd.to_datetime('1/1/2015 01:00') t2 = pd.to_datetime('1/1/2015 03:30') print pd.Timedelta(t2...

Answer : NetworkX expects a square matrix (of nodes and edges), perhaps* you want to pass it: In [11]: df2 = pd.concat([df, df.T]).fillna(0) Note: It's important that the index and columns are in the same order! In [12]: df2 = df2.reindex(df2.columns) In [13]: df2 Out[13]: Bar Bat Baz Foo Loc 1 Loc 2 Loc 3 Loc 4 Loc 5 Loc 6 Loc 7 Quux Bar 0 0 0 0 0 0 1 1 0 1 1 0 Bat 0 0 0 0 0 0 1 0 0 1 0 0 Baz 0 0 0 0 0 0 1 0 0 0 0 0 Foo 0 0 0 0 0 0 1 1 0 0 0 0 Loc 1 0 0 0 0 0 0 0 0 0 0 0 1 Loc 2 0 0 0 0 0 0 0 0 0 0 0 0 Loc 3 1 1 1 1 0 0 0 0 0 0 0 0 Loc 4 1 0 0 1 0 0 0 0 0 0 ...

Answer : Pandas concat takes a list of dataframes. If you can generate a list of dataframes with your looping function, once you are finished you can concatenate the list together: data_day_list = [] for i, day in enumerate(list_day): data_day = df[df.day==day] data_day_list.append(data_day) final_data_day = pd.concat(data_day_list) Exhausting a generator is more elegant (if not more efficient) than appending to a list. For example: def yielder(df, list_day): for i, day in enumerate(list_day): yield df[df['day'] == day] final_data_day = pd.concat(list(yielder(df, list_day)) Appending or concatenating pd.DataFrame s is slow. You can use a list in the interim and then create the final pd.DataFrame at the end with pd.DataFrame.from_records() e.g.: interim_list = [] for i,(k,g) in enumerate(df.groupby(['[*name of your date column here*'])): if i % 1000 == 0 and i != 0: print('iteration: {}'.format(i)) # just tells you where you are i...

Answer : You need nunique : df = df.groupby('domain')['ID'].nunique() print (df) domain 'facebook.com' 1 'google.com' 1 'twitter.com' 2 'vk.com' 3 Name: ID, dtype: int64 If you need to strip ' characters: df = df.ID.groupby([df.domain.str.strip("'")]).nunique() print (df) domain facebook.com 1 google.com 1 twitter.com 2 vk.com 3 Name: ID, dtype: int64 Or as Jon Clements commented: df.groupby(df.domain.str.strip("'"))['ID'].nunique() You can retain the column name like this: df = df.groupby(by='domain', as_index=False).agg({'ID': pd.Series.nunique}) print(df) domain ID 0 fb 1 1 ggl 1 2 twitter 2 3 vk 3 The difference is that nunique() returns a Series and agg() returns a DataFrame. Generally to count distinct values in single column, you can use Series.value_counts : df.domain.value_counts() #'vk.com...

Answer : The seaborn equivalent of df.boxplot() is sns.boxplot(x="variable", y="value", data=pd.melt(df)) Complete example: import numpy as np; np.random.seed(42) import pandas as pd import matplotlib.pyplot as plt import seaborn as sns df = pd.DataFrame(data = np.random.random(size=(4,4)), columns = ['A','B','C','D']) sns.boxplot(x="variable", y="value", data=pd.melt(df)) plt.show() This works because pd.melt converts a wide-form dataframe A B C D 0 0.374540 0.950714 0.731994 0.598658 1 0.156019 0.155995 0.058084 0.866176 2 0.601115 0.708073 0.020584 0.969910 3 0.832443 0.212339 0.181825 0.183405 to long-form variable value 0 A 0.374540 1 A 0.156019 2 A 0.601115 3 A 0.832443 4 B 0.950714 5 B 0.155995 6 B 0.708073 7 B 0.212339 8 C 0.731994 9 C ...