LVM is static component, but is not cluster aware.
It means that, when system starts and recognized LVM partitions and volumes,
it dont have any dynamic internal state (it just map one
devices onto the other devices), so I don't see any reasons, why it can not
work with cluster. If you create LVM
volumes on one node, run lvm scan (vgscan, as I remember) on others, then
restart ALL nodes and verify, that they all recognize all LVM objects - you
can run RAC cluster safely, all nodes will be using the same volume mapping.
(It is not true for RAID, for example, because RAID have internal dynamic
state. Fortunately, LVM have not internal state when it runs just
logical volumes to the disks mapping and you use raw devices on it /I am not
100% sure, through/).
It can be a little tricky (de facto, you need to import forein volumes, and
I dont remember how it is done in LVM, but
as I remember, you just need to start lvm and it scans disks and find
everything), but it's one time job.
Then you can run RAC until you want to change anything on LVM. If you change
(add volume on one node), other nodes will not see
this change until you rescan LVM on them (and I am not sure, if they
recognzie LVM change on the fly, without system restart).
So, except if you experiment and find a right sequence of LVM commands,
allowing you to rescan volumes and recognize change on the fly,
you must, instead:
- before any LVM change, shutdown all nodes except one;
- on this node, make all changes but don't change ORACLE setting in this
phase;
- start other nodes and verify, that they can see your change (new volume
size for example);
- now DBA can change Oracle setting (resize tablefile for example).
In case of ASM, it is much simpler - all I need to do, if I want to add
space, is
- create LUN on SAN system
- rescan scsi (I use iSCSI so it's pretty simple - just echo 1 >
...../rescan files) on all servers so that servers see new disk;
- add new raw device if necessary (I create them as a symlinks in
/dev/rawnames)
- add disk into disk group using dbca or EM.
I dont need to shutdown nodes.
I did not experiment, how lvm behave on shared disks (but I used LVM @
FireWire disk on RQC cluster few years ago, and it worked well in the lab).
May be, you wil be able to rescan disks by vgscan command. Problem is that,
if you use LVM and non - standard configuration (manually run vgscan for
example), it is difficult to be sure, that LVM disks will be corerctly
recognized after system reboot (for example). So, as I said - experiment
(having running RAC cluster) and make detailed policy for
- adding new disk into LVM
- adding space into LVM volume
- creating new LVM group
- creating new LVM volume
on your cluster. if you succeed - you can sleep at night and don't afraid
that your DB break after node restart or volume resizing.
PS. Very serious problem is with LUN resizing. Most SAN systems allows to
resize LUN;'s on the fly. ASM allows to resize disk groups
if it ca see that size changed. Unfortunately, Linux dont allow applications
to see disk size change, if disk is used by anyone, so you can't just resize
LUN on SAN system and then resize it in Oracle. Fortunately, you can just
add new disk into ASM disk group. New cluster-aware lvm or evms are going to
resolve it, but (aside of new and complicated evms in SELS10 ) it don't
exist in SLES9 (and Novell will not do it in SLES9, as we can see - they are
goiing to push SLES10 instead of improving very stable and very compatible
SLES9).
----- Original Message -----
From: "C'est Pierre" <cestpierre@(protected)>
To: "Alexei_Roudnev" <Alexei_Roudnev@(protected)>
Cc: "Didier Boiteux" <dboiteux@(protected)>;
<suse-sles-e@(protected)>
Sent: Friday, October 13, 2006 6:18 PM
Subject: Re: [suse-oracle] Re: [suse-sles-e] Re: [suse-oracle] Re:
[suse-sles-e] Re: [suse-oracle] Raw Devices
Hi Alexei!
Thanks for the reply!
Could you be a little more detailed here? I don't think I got the
whole picture perfectly. I know lvm isn't cluster-aware, but maybe I
missed a part of it. Do you mean, if I add an LV while oracle's nodes
are on, something bad might happen? if so, what type of madness should
we expect of it? what should be avoided at all costs? should Vgs on
both nodes (2 node rac cluster) go through vgchange -a y ?
Thanks
Pierre
On 10/14/06, Alexei_Roudnev <Alexei_Roudnev@(protected):
> Is he awared, that LVM is not cluster-aware and so he must be extremely
> careful (and have a high chance to make a critical error and damage a
> database)
> using LVM with RAC cluster? (It means - shutdown all nodes but one before
> ANY LVM change, and verify, that LVM change recognized on all nodes, after
> starting them)
>
> (I'd better use symlinks to raw devices, and manage raw devices as a
LUN';s
> on SAN system, if I am too awared of ASM stability. Raw over LVM is not a
> bad idea in general, but not in case of RAC and distributed /DBA and
> sysadmin/ responsibility).
>
> But if you make working change policy for LVM (how to add / change LVM),
> then _why not? It can work_.
>
>
> ----- Original Message -----
> From: "C'est Pierre" <cestpierre@(protected)>
> To: "Didier Boiteux" <dboiteux@(protected)>
> Cc: <suse-oracle@(protected)>
> Sent: Friday, October 13, 2006 3:05 AM
> Subject: [suse-oracle] Re: [suse-sles-e] Re: [suse-oracle] Re:
[suse-sles-e]
> Re: [suse-oracle] Raw Devices
>
>
> Thanks to all who replied to my questions!
>
> Our dba decided to go with LVM until ASM is more "stable", he says.
>
> Thanks to all!
> Pierre
>
>
> On 10/11/06, Didier Boiteux <dboiteux@(protected):
> > Pierre,
> >
> > The SLES9 kernel included in SP3 was indeed 2.6.5-7.244 on 12th Dec
2005.
> > (See http://www.novell.com/products/server9/packages.html
> > or from the INDEX.gz on the SP3 CD)
> >
> > Many news kernel patches have appeared since then.
> > Here is a useful link to find easily all the rpm patches (like for the
> > kernel):
> >
> > - For the "core" packages:
> > For x86_64: -> for the kernel ones
> > https://you.novell.com/update/x86_64/update/SUSE-CORE/9/rpm/x86_64/
> > - For the specific SLES
> > For x86_64:
> > https://you.novell.com/update/x86_64/update/SUSE-SLES/9/rpm/x86_64/
> >
> > You´ll find the 2.6.5-7.257 there.
> > (In case you wonder, the difference between the core and SLEs pacakges
> comes
> > from the United Linux experience so that the core packages would be
shared
> by
> > all United Linux distributions...)
> >
> >
> > You can also look at the Patch Support DataBase (PSDB) at
> > http://support.novell.com/linux/psdb/byproduct.html
> > and look for SLEs9 x86_64, and then search for "kernel", but it is
> slightly
> > longer...
> >
> > For example, at present, the latest is 2.6.5-7.282 (patch-11193).
> >
> >
> > -+-
> > Didier
> >
> >
> > On Tuesday 10 October 2006 12:09, C'est Pierre wrote:
> > > Hello all!
> > >
> > > Sorry for the late reply, but I had a few other tasks to solve in
> between.
> > > We're going to try the lvm + raw approach. Any issues?
> > >
> > > In case of failure, we'll resort to OCFSv2. I've been to Oracle's OCFS
> > > page and they say we require SLES 9 SP3's kernel 2.6.5-7.257. However,
> > > my SP3 iso, downloaded from Novell's website, has these files:
> > >
> > >
>
./x86_64/update/SUSE-CORE/9/rpm/x86_64/kernel-source-2.6.5-7.244.x86_64.rpm
> > >
> ./x86_64/update/SUSE-CORE/9/rpm/x86_64/kernel-syms-2.6.5-7.244.x86_64.rpm
> > >
>
./x86_64/update/SUSE-CORE/9/rpm/x86_64/kernel-default-2.6.5-7.244.x86_64.rp
> > >m
> ./x86_64/update/SUSE-CORE/9/rpm/x86_64/kernel-smp-2.6.5-7.244.x86_64.rpm
> > >
> > > Any ideas?
> > >
> > > Thanks,
> > > Pierre
> > >
> > > On 10/9/06, Alexei_Roudnev <Alexei_Roudnev@(protected):
> > > > I think, that your DBA tried to do a bad thing - change disk size
on
> SAN
> > > > system and then resize ASM device., It don';t work because Linux
don't
> > > > see new size if disk is mounted by any way.
> > > >
> > > > You should not
> > > > - use ASMLib, it is a piece of garbage.
> > > > - don't try to resize disk when ASM device is full.
> > > >
> > > > You should, instead, ADD new disk into the group, when previous one
is
> > > > filled in (and system will redistribute ASM files), OR add new
> > > > ASM disk group and assign new datafile into the tablespace.
> > > >
> > > > OCFSv2 is a controvercial thing - it is fine when it work, buit it
> adds
> > > > an element of instability. I;d better use NFS for backups, but we
are
> > > > runnimng OCFSv2 in the lab for low-performance file systems (archive
> logs
> > > > and backups) and it works fine (if we have not infrastructure
> failures).
> > > >
> > > >
> > > >
> > > > ----- Original Message -----
> > > > From: "C'est Pierre" <cestpierre@(protected)>
> > > > To: "Fabrizio Magni" <fabrizio.magni@(protected)>
> > > > Cc: <suse-oracle@(protected)>
> > > > Sent: Monday, October 09, 2006 8:41 AM
> > > > Subject: [suse-sles-e] Re: [suse-oracle] Raw Devices
> > > >
> > > > > Hi Fabrizio,
> > > > >
> > > > > We skipped the option of using ASMLib on this cluster. Our dba
said
> it
> > > > > had a limitation (or bug) when a tablespace (or was it raw
device?)
> > > > > filled up, they couldn't extend the size anymore. This is a
> migration
> > > > > database, so they'll need lots more space than we already have in
> the
> > > > > long run.
> > > > >
> > > > > We're using RAC and raw devices only, from what he said. An
> > > > > alternative in mind is OCFS.
> > > > >
> > > > > Any thoughts?
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Pierre
> > > > >
> > > > > On 10/9/06, Fabrizio Magni <fabrizio.magni@(protected):
> > > > > > Hi Pierre,
> > > > > > just personal curiosity: what's the advantage to have 47
partition
> on
> > > >
> > > > the
> > > >
> > > > > > same disk?
> > > > > > If you are still using the ASM then you can simply use a single
> > > >
> > > > partition
> > > >
> > > > > > (or device).
> > > > > > Tablespaces are already divided inside that device optimizing
the
> > > > > > space usage.
> > > > > >
> > > > > > Regards
> > > > > > Fabrizio
> > > > > >
> > > > > > On 10/9/06, C'est Pierre <cestpierre@(protected):
> > > > > > > Hello all,
> > > > > > >
> > > > > > > First, I am sys-administrator, not a DBA, so excuse me if I am
> no
> > > > > > > precise with some tech. aspects of Oracle.
> > > > > > >
> > > > > > > I am in the process of installing a new RAC cluster. Our
> previous
> > > > > > > one, went smoothly, but we used ASMLib, which we aren't using
> now.
> > > > > > > Problem: I created 47 partitions on a scsi disk presented to
the
> > > > > > > server thru a fibre channel card. These are sda1 thru sda47.
> > > > > > > However, only the 15 first (except one, which is the extended
> one -
> > > > > > > sda4) are usable.
> > > > > > >
> > > > > > > I wrote the /etc/raw, mapping sda's to raw's just as the dba
> > > > > > > sugested for organizational purposes (they aren't sequential,
> e.g
> > > > > > > raw51 maps to sda28).
> > > > > > >
> > > > > > > I then made this line of bash to create all device nods that
> > > > > > > weren't there (and even those that were...just in case):
> > > > > > >
> > > > > > > for i in `seq -f %g 1 47`; do echo mknod sda$i b 8 $i ;
done
> > > > > > >
> > > > > > > Problem: I can't access past sda16, I get this error:
> > > > > > >
> > > > > > > dd: opening `/dev/sda16': No such device or address
> > > > > > >
> > > > > > > when I look at dmesg and /proc/partitions, I only get to see
the
> > > > > > > first 15 partitions there (or 16, if you count with 'sda')
> > > > > > >
> > > > > > > sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 sda9 sda10
sda11
> > > > > > > sda12 sda13 sda14 sda15 >
> > > > > > >
> > > > > > > I've googled for this problem and it seems there's a
limitation
> on
> > > > > > > scsi disks of 16 partitions per device. Is this true? if so,
> what
> > > > > > > is the solution for this problem? how can I make 47 raw
> partitions?
> > > > > > >
> > > > > > > I can still ask our storage administrator to divide this disk
> into
> > > > > > > several disks and then we will group partitions 16 partitions
on
> > > > > > > each, but this isn't a good solution. Another alternative
> solution
> > > > > > > which our dba presented, was adding ocfs 2 support and make it
> > > > > > > dance with the devil by the pale moonlight ;-)
> > > > > > >
> > > > > > > Let me know your thoughts and especially solutions!!
> > > > > > >
> > > > > > > Also, your thoughts on ocfs or not as a side note would be
> > > > > > > apreciated.
> > > > > > >
> > > > > > > Thank you
> > > > > > >
> > > > > > > Pierre
> > > > > > >
> > > > > > > P.S: I'm not french but I love french bread! ;)
> > > > > > >
> > > > > > > --
> > > > > > > To unsubscribe, email: suse-oracle-unsubscribe@(protected)
> > > > > > > For additional commands, email: suse-oracle-help@(protected)
> > > > > > > Please see http://www.suse.com/oracle/ before posting
> > > > >
> > > >
> > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: suse-sles-e-unsubscribe@(protected)
> > > > > For additional commands, e-mail: suse-sles-e-help@(protected)
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: suse-sles-e-unsubscribe@(protected)
> > For additional commands, e-mail: suse-sles-e-help@(protected)
> >
> >
>
> --
> To unsubscribe, email: suse-oracle-unsubscribe@(protected)
> For additional commands, email: suse-oracle-help@(protected)
> Please see http://www.suse.com/oracle/ before posting
>
>
>
--
To unsubscribe, email: suse-oracle-unsubscribe@(protected)
For additional commands, email: suse-oracle-help@(protected)
Please see http://www.suse.com/oracle/ before posting
