Hi Alexei,
Wow! That was long! Thanks ;)
I am now feeling a lot better. I did the vgscan and lvscan on the 2nd
node after I lvcreate'd a few volumes on the first one, then I did a
script to update /etc/raw and ran /etc/rc.d/raw start, the only thing
that I had to do afterwards was lvchange -a y on the newly created
LVs.. everything seemed right. DBA didn't scream as well.
As for the system, we're currently using an HP XP1024, which adds 2
lun's to our server and we're also using HP QLogic's driver, with
failover built-in.
As far as DBA told me, he's afraid of using ASM due to problem with
resizing tablespaces. Our luns have been dimensioned to the point
where we'll not need more space and we're also not using the entire
available space, so, if we'll ever need to add more space to a
tablespace, we'll just add another LV with the needed space (or so, I
guess, as I am not a DBA). I am also assuming the dba didn't created
the tablespaces with the 'auto-resize' or 'auto-grow' option or
something like that (I am newbie in oracle, created today my first
tablespace in a test db, as I am looking forward to learn more about
oracle to be able to think of valid solutions before our dba comes up
with its own and be able to argue with him)
If I am wrong anywhere, feel free to enlighten me!
Your info was very much appreciated! Thank you for your time and patience :-)
Pierre
On 10/14/06, Alexei_Roudnev <Alexei_Roudnev@(protected):
> LVM is static component, but is not cluster aware.
>
> It means that, when system starts and recognized LVM partitions and volumes,
> it dont have any dynamic internal state (it just map one
> devices onto the other devices), so I don't see any reasons, why it can not
> work with cluster. If you create LVM
> volumes on one node, run lvm scan (vgscan, as I remember) on others, then
> restart ALL nodes and verify, that they all recognize all LVM objects - you
> can run RAC cluster safely, all nodes will be using the same volume mapping.
>
> (It is not true for RAID, for example, because RAID have internal dynamic
> state. Fortunately, LVM have not internal state when it runs just
> logical volumes to the disks mapping and you use raw devices on it /I am not
> 100% sure, through/).
>
> It can be a little tricky (de facto, you need to import forein volumes, and
> I dont remember how it is done in LVM, but
> as I remember, you just need to start lvm and it scans disks and find
> everything), but it's one time job.
>
> Then you can run RAC until you want to change anything on LVM. If you change
> (add volume on one node), other nodes will not see
> this change until you rescan LVM on them (and I am not sure, if they
> recognzie LVM change on the fly, without system restart).
>
> So, except if you experiment and find a right sequence of LVM commands,
> allowing you to rescan volumes and recognize change on the fly,
> you must, instead:
> - before any LVM change, shutdown all nodes except one;
> - on this node, make all changes but don't change ORACLE setting in this
> phase;
> - start other nodes and verify, that they can see your change (new volume
> size for example);
> - now DBA can change Oracle setting (resize tablefile for example).
>
> In case of ASM, it is much simpler - all I need to do, if I want to add
> space, is
> - create LUN on SAN system
> - rescan scsi (I use iSCSI so it's pretty simple - just echo 1 >
> ...../rescan files) on all servers so that servers see new disk;
> - add new raw device if necessary (I create them as a symlinks in
> /dev/rawnames)
> - add disk into disk group using dbca or EM.
>
> I dont need to shutdown nodes.
>
> I did not experiment, how lvm behave on shared disks (but I used LVM @
> FireWire disk on RQC cluster few years ago, and it worked well in the lab).
> May be, you wil be able to rescan disks by vgscan command. Problem is that,
> if you use LVM and non - standard configuration (manually run vgscan for
> example), it is difficult to be sure, that LVM disks will be corerctly
> recognized after system reboot (for example). So, as I said - experiment
> (having running RAC cluster) and make detailed policy for
> - adding new disk into LVM
> - adding space into LVM volume
> - creating new LVM group
> - creating new LVM volume
> on your cluster. if you succeed - you can sleep at night and don't afraid
> that your DB break after node restart or volume resizing.
>
> PS. Very serious problem is with LUN resizing. Most SAN systems allows to
> resize LUN;'s on the fly. ASM allows to resize disk groups
> if it ca see that size changed. Unfortunately, Linux dont allow applications
> to see disk size change, if disk is used by anyone, so you can't just resize
> LUN on SAN system and then resize it in Oracle. Fortunately, you can just
> add new disk into ASM disk group. New cluster-aware lvm or evms are going to
> resolve it, but (aside of new and complicated evms in SELS10 ) it don't
> exist in SLES9 (and Novell will not do it in SLES9, as we can see - they are
> goiing to push SLES10 instead of improving very stable and very compatible
> SLES9).
>
>
> ----- Original Message -----
> From: "C'est Pierre" <cestpierre@(protected)>
> To: "Alexei_Roudnev" <Alexei_Roudnev@(protected)>
> Cc: "Didier Boiteux" <dboiteux@(protected)>;
> <suse-sles-e@(protected)>
> Sent: Friday, October 13, 2006 6:18 PM
> Subject: Re: [suse-oracle] Re: [suse-sles-e] Re: [suse-oracle] Re:
> [suse-sles-e] Re: [suse-oracle] Raw Devices
>
>
> Hi Alexei!
>
> Thanks for the reply!
>
> Could you be a little more detailed here? I don't think I got the
> whole picture perfectly. I know lvm isn't cluster-aware, but maybe I
> missed a part of it. Do you mean, if I add an LV while oracle's nodes
> are on, something bad might happen? if so, what type of madness should
> we expect of it? what should be avoided at all costs? should Vgs on
> both nodes (2 node rac cluster) go through vgchange -a y ?
>
> Thanks
> Pierre
>
> On 10/14/06, Alexei_Roudnev <Alexei_Roudnev@(protected):
> > Is he awared, that LVM is not cluster-aware and so he must be extremely
> > careful (and have a high chance to make a critical error and damage a
> > database)
> > using LVM with RAC cluster? (It means - shutdown all nodes but one before
> > ANY LVM change, and verify, that LVM change recognized on all nodes, after
> > starting them)
> >
> > (I'd better use symlinks to raw devices, and manage raw devices as a
> LUN';s
> > on SAN system, if I am too awared of ASM stability. Raw over LVM is not a
> > bad idea in general, but not in case of RAC and distributed /DBA and
> > sysadmin/ responsibility).
> >
> > But if you make working change policy for LVM (how to add / change LVM),
> > then _why not? It can work_.
> >
> >
> > ----- Original Message -----
> > From: "C'est Pierre" <cestpierre@(protected)>
> > To: "Didier Boiteux" <dboiteux@(protected)>
> > Cc: <suse-oracle@(protected)>
> > Sent: Friday, October 13, 2006 3:05 AM
> > Subject: [suse-oracle] Re: [suse-sles-e] Re: [suse-oracle] Re:
> [suse-sles-e]
> > Re: [suse-oracle] Raw Devices
> >
> >
> > Thanks to all who replied to my questions!
> >
> > Our dba decided to go with LVM until ASM is more "stable", he says.
> >
> > Thanks to all!
> > Pierre
> >
> >
> > On 10/11/06, Didier Boiteux <dboiteux@(protected):
> > > Pierre,
> > >
> > > The SLES9 kernel included in SP3 was indeed 2.6.5-7.244 on 12th Dec
> 2005.
> > > (See http://www.novell.com/products/server9/packages.html
> > > or from the INDEX.gz on the SP3 CD)
> > >
> > > Many news kernel patches have appeared since then.
> > > Here is a useful link to find easily all the rpm patches (like for the
> > > kernel):
> > >
> > > - For the "core" packages:
> > > For x86_64: -> for the kernel ones
> > > https://you.novell.com/update/x86_64/update/SUSE-CORE/9/rpm/x86_64/
> > > - For the specific SLES
> > > For x86_64:
> > > https://you.novell.com/update/x86_64/update/SUSE-SLES/9/rpm/x86_64/
> > >
> > > You´ll find the 2.6.5-7.257 there.
> > > (In case you wonder, the difference between the core and SLEs pacakges
> > comes
> > > from the United Linux experience so that the core packages would be
> shared
> > by
> > > all United Linux distributions...)
> > >
> > >
> > > You can also look at the Patch Support DataBase (PSDB) at
> > > http://support.novell.com/linux/psdb/byproduct.html
> > > and look for SLEs9 x86_64, and then search for "kernel", but it is
> > slightly
> > > longer...
> > >
> > > For example, at present, the latest is 2.6.5-7.282 (patch-11193).
> > >
> > >
> > > -+-
> > > Didier
> > >
> > >
> > > On Tuesday 10 October 2006 12:09, C'est Pierre wrote:
> > > > Hello all!
> > > >
> > > > Sorry for the late reply, but I had a few other tasks to solve in
> > between.
> > > > We're going to try the lvm + raw approach. Any issues?
> > > >
> > > > In case of failure, we'll resort to OCFSv2. I've been to Oracle's OCFS
> > > > page and they say we require SLES 9 SP3's kernel 2.6.5-7.257. However,
> > > > my SP3 iso, downloaded from Novell's website, has these files:
> > > >
> > > >
> >
> ./x86_64/update/SUSE-CORE/9/rpm/x86_64/kernel-source-2.6.5-7.244.x86_64.rpm
> > > >
> > ./x86_64/update/SUSE-CORE/9/rpm/x86_64/kernel-syms-2.6.5-7.244.x86_64.rpm
> > > >
> >
> ./x86_64/update/SUSE-CORE/9/rpm/x86_64/kernel-default-2.6.5-7.244.x86_64.rp
> > > >m
> > ./x86_64/update/SUSE-CORE/9/rpm/x86_64/kernel-smp-2.6.5-7.244.x86_64.rpm
> > > >
> > > > Any ideas?
> > > >
> > > > Thanks,
> > > > Pierre
> > > >
> > > > On 10/9/06, Alexei_Roudnev <Alexei_Roudnev@(protected):
> > > > > I think, that your DBA tried to do a bad thing - change disk size
> on
> > SAN
> > > > > system and then resize ASM device., It don';t work because Linux
> don't
> > > > > see new size if disk is mounted by any way.
> > > > >
> > > > > You should not
> > > > > - use ASMLib, it is a piece of garbage.
> > > > > - don't try to resize disk when ASM device is full.
> > > > >
> > > > > You should, instead, ADD new disk into the group, when previous one
> is
> > > > > filled in (and system will redistribute ASM files), OR add new
> > > > > ASM disk group and assign new datafile into the tablespace.
> > > > >
> > > > > OCFSv2 is a controvercial thing - it is fine when it work, buit it
> > adds
> > > > > an element of instability. I;d better use NFS for backups, but we
> are
> > > > > runnimng OCFSv2 in the lab for low-performance file systems (archive
> > logs
> > > > > and backups) and it works fine (if we have not infrastructure
> > failures).
> > > > >
> > > > >
> > > > >
> > > > > ----- Original Message -----
> > > > > From: "C'est Pierre" <cestpierre@(protected)>
> > > > > To: "Fabrizio Magni" <fabrizio.magni@(protected)>
> > > > > Cc: <suse-oracle@(protected)>
> > > > > Sent: Monday, October 09, 2006 8:41 AM
> > > > > Subject: [suse-sles-e] Re: [suse-oracle] Raw Devices
> > > > >
> > > > > > Hi Fabrizio,
> > > > > >
> > > > > > We skipped the option of using ASMLib on this cluster. Our dba
> said
> > it
> > > > > > had a limitation (or bug) when a tablespace (or was it raw
> device?)
> > > > > > filled up, they couldn't extend the size anymore. This is a
> > migration
> > > > > > database, so they'll need lots more space than we already have in
> > the
> > > > > > long run.
> > > > > >
> > > > > > We're using RAC and raw devices only, from what he said. An
> > > > > > alternative in mind is OCFS.
> > > > > >
> > > > > > Any thoughts?
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Pierre
> > > > > >
> > > > > > On 10/9/06, Fabrizio Magni <fabrizio.magni@(protected):
> > > > > > > Hi Pierre,
> > > > > > > just personal curiosity: what's the advantage to have 47
> partition
> > on
> > > > >
> > > > > the
> > > > >
> > > > > > > same disk?
> > > > > > > If you are still using the ASM then you can simply use a single
> > > > >
> > > > > partition
> > > > >
> > > > > > > (or device).
> > > > > > > Tablespaces are already divided inside that device optimizing
> the
> > > > > > > space usage.
> > > > > > >
> > > > > > > Regards
> > > > > > > Fabrizio
> > > > > > >
> > > > > > > On 10/9/06, C'est Pierre <cestpierre@(protected):
> > > > > > > > Hello all,
> > > > > > > >
> > > > > > > > First, I am sys-administrator, not a DBA, so excuse me if I am
> > no
> > > > > > > > precise with some tech. aspects of Oracle.
> > > > > > > >
> > > > > > > > I am in the process of installing a new RAC cluster. Our
> > previous
> > > > > > > > one, went smoothly, but we used ASMLib, which we aren't using
> > now.
> > > > > > > > Problem: I created 47 partitions on a scsi disk presented to
> the
> > > > > > > > server thru a fibre channel card. These are sda1 thru sda47.
> > > > > > > > However, only the 15 first (except one, which is the extended
> > one -
> > > > > > > > sda4) are usable.
> > > > > > > >
> > > > > > > > I wrote the /etc/raw, mapping sda's to raw's just as the dba
> > > > > > > > sugested for organizational purposes (they aren't sequential,
> > e.g
> > > > > > > > raw51 maps to sda28).
> > > > > > > >
> > > > > > > > I then made this line of bash to create all device nods that
> > > > > > > > weren't there (and even those that were...just in case):
> > > > > > > >
> > > > > > > > for i in `seq -f %g 1 47`; do echo mknod sda$i b 8 $i ;
> done
> > > > > > > >
> > > > > > > > Problem: I can't access past sda16, I get this error:
> > > > > > > >
> > > > > > > > dd: opening `/dev/sda16': No such device or address
> > > > > > > >
> > > > > > > > when I look at dmesg and /proc/partitions, I only get to see
> the
> > > > > > > > first 15 partitions there (or 16, if you count with 'sda')
> > > > > > > >
> > > > > > > > sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 sda9 sda10
> sda11
> > > > > > > > sda12 sda13 sda14 sda15 >
> > > > > > > >
> > > > > > > > I've googled for this problem and it seems there's a
> limitation
> > on
> > > > > > > > scsi disks of 16 partitions per device. Is this true? if so,
> > what
> > > > > > > > is the solution for this problem? how can I make 47 raw
> > partitions?
> > > > > > > >
> > > > > > > > I can still ask our storage administrator to divide this disk
> > into
> > > > > > > > several disks and then we will group partitions 16 partitions
> on
> > > > > > > > each, but this isn't a good solution. Another alternative
> > solution
> > > > > > > > which our dba presented, was adding ocfs 2 support and make it
> > > > > > > > dance with the devil by the pale moonlight ;-)
> > > > > > > >
> > > > > > > > Let me know your thoughts and especially solutions!!
> > > > > > > >
> > > > > > > > Also, your thoughts on ocfs or not as a side note would be
> > > > > > > > apreciated.
> > > > > > > >
> > > > > > > > Thank you
> > > > > > > >
> > > > > > > > Pierre
> > > > > > > >
> > > > > > > > P.S: I'm not french but I love french bread! ;)
> > > > > > > >
> > > > > > > > --
> > > > > > > > To unsubscribe, email: suse-oracle-unsubscribe@(protected)
> > > > > > > > For additional commands, email: suse-oracle-help@(protected)
> > > > > > > > Please see http://www.suse.com/oracle/ before posting
> > > > > >
> > > > >
> > > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: suse-sles-e-unsubscribe@(protected)
> > > > > > For additional commands, e-mail: suse-sles-e-help@(protected)
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: suse-sles-e-unsubscribe@(protected)
> > > For additional commands, e-mail: suse-sles-e-help@(protected)
> > >
> > >
> >
> > --
> > To unsubscribe, email: suse-oracle-unsubscribe@(protected)
> > For additional commands, email: suse-oracle-help@(protected)
> > Please see http://www.suse.com/oracle/ before posting
> >
> >
> >
>
>
--
To unsubscribe, email: suse-oracle-unsubscribe@(protected)
For additional commands, email: suse-oracle-help@(protected)
Please see http://www.suse.com/oracle/ before posting
